Reinforcement World Model Learning for LLM-based Agents
Large language models (LLMs) have achieved strong performance in language-centric tasks. However, in agentic settings, LLMs often struggle to anticipate action consequences and adapt to environment dynamics, highlighting the need for world-modeling capabilities in LLM-based agents. We propose Reinforcement World Model Learning (RWML), a self-supervised method that learns action-conditioned world models for LLM-based agents on textual states using sim-to-real gap rewards. Our method aligns simulated next states produced by the model with realized next states observed from the environment, encouraging consistency between internal world simulations and actual environment dynamics in a pre-trained embedding space. Unlike next-state token prediction, which prioritizes token-level fidelity (i.e., reproducing exact wording) over semantic equivalence and can lead to model collapse, our method provides a more robust training signal and is empirically less susceptible to reward hacking than LLM-as-a-judge. We evaluate our method on ALFWorld and τ^2 Bench and observe significant gains over the base model, despite being entirely self-supervised. When combined with task-success rewards, our method outperforms direct task-success reward RL by 6.9 and 5.7 points on ALFWorld and τ^2 Bench respectively, while matching the performance of expert-data training.
69855bf84ad556f294b7eb1e
Xiao Yu
69855bf84ad556f294b7eb1f
Baolin Peng
69855bf84ad556f294b7eb20
Ruize Xu
69855bf84ad556f294b7eb21
Yelong Shen
69855bf84ad556f294b7eb22
Pengcheng He
69855bf84ad556f294b7eb23
Suman Nath
69855bf84ad556f294b7eb24
Nikhil Singh
69855bf84ad556f294b7eb25
Jiangfeng Gao
69855bf84ad556f294b7eb26
Zhou Yu
Reinforcement World Model Learning for LLM-based Agents
Large language models (LLMs) have achieved strong performance in language-centric tasks. However, in agentic settings, LLMs often struggle to anticipate action consequences and adapt to environment dynamics, highlighting the need for world-modeling capabilities in LLM-based agents. We propose Reinforcement World Model Learning (RWML), a self-supervised method that learns action-conditioned world models for LLM-based agents on textual states using sim-to-real gap rewards. Our method aligns simulated next states produced by the model with realized next states observed from the environment, encouraging consistency between internal world simulations and actual environment dynamics in a pre-trained embedding space. Unlike next-state token prediction, which prioritizes token-level fidelity (i.e., reproducing exact wording) over semantic equivalence and can lead to model collapse, our method provides a more robust training signal and is empirically less susceptible to reward hacking than LLM-as-a-judge. We evaluate our method on ALFWorld and τ^2 Bench and observe significant gains over the base model, despite being entirely self-supervised. When combined with task-success rewards, our method outperforms direct task-success reward RL by 6.9 and 5.7 points on ALFWorld and τ^2 Bench respectively, while matching the performance of expert-data training.
Reinforcement World Model Learning enables LLM-based agents to better anticipate action consequences and adapt to environment dynamics through self-supervised training that aligns simulated and real-world state transitions in embedding space.
world-modeling reinforcement learning action-conditioned world models sim-to-real gap rewards next-state token prediction reward hacking task-success rewards embedding space agent-based systems