Scaling Agent Learning via Experience Synthesis
While reinforcement learning (RL) can empower large language model (LLM)
agents by enabling self-improvement through interaction, its practical adoption
remains challenging due to costly rollouts, limited task diversity, unreliable
reward signals, and infrastructure complexity, all of which obstruct the
collection of scalable experience data. To address these challenges, we
introduce DreamGym, the first unified framework designed to synthesize diverse
experiences with scalability in mind to enable effective online RL training for
autonomous agents. Rather than relying on expensive real-environment rollouts,
DreamGym distills environment dynamics into a reasoning-based experience model
that derives consistent state transitions and feedback signals through
step-by-step reasoning, enabling scalable agent rollout collection for RL. To
improve the stability and quality of transitions, DreamGym leverages an
experience replay buffer initialized with offline real-world data and
continuously enriched with fresh interactions to actively support agent
training. To improve knowledge acquisition, DreamGym adaptively generates new
tasks that challenge the current agent policy, enabling more effective online
curriculum learning. Experiments across diverse environments and agent
backbones demonstrate that DreamGym substantially improves RL training, both in
fully synthetic settings and in sim-to-real transfer scenarios. On non-RL-ready
tasks like WebArena, DreamGym outperforms all baselines by over 30%. And in
RL-ready but costly settings, it matches GRPO and PPO performance using only
synthetic interactions. When transferring a policy trained purely on synthetic
experiences to real-environment RL, DreamGym yields significant additional
performance gains while requiring far fewer real-world interactions, providing
a scalable warm-start strategy for general-purpose RL.
690d51e4ad2597bf6c464c52
Zhaorun Chen
690d51e4ad2597bf6c464c53
Zhuokai Zhao
claimed_verified
2025-11-10T09:32:09.099Z
63af25605fe9db73f67a0fb7
https://cdn-avatars.huggingface.co/v1/production/uploads/63af25605fe9db73f67a0fb7/mcQHbTdJVxi2GdHzRB9ad.jpeg
Zhuokai Zhao
zhuokai
user
zhuokai
690d51e4ad2597bf6c464c54
Kai Zhang
claimed_verified
2025-11-07T10:30:52.478Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
drogozhang
user
drogozhang
690d51e4ad2597bf6c464c55
Bo Liu
claimed_verified
2025-11-07T10:30:56.570Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
Benjamin-eecs
user
Benjamin-eecs
690d51e4ad2597bf6c464c56
Qi Qi
690d51e4ad2597bf6c464c57
Yifan Wu
claimed_verified
2025-11-07T10:30:54.351Z
64641a2938083255f6769953
/avatars/a4117357703607bd7b290dc2975acbef.svg
Yifan Wu
yfwu
user
yfwu
690d51e4ad2597bf6c464c58
Tarun Kalluri
690d51e4ad2597bf6c464c59
Sara Cao
690d51e4ad2597bf6c464c5a
Yuanhao Xiong
690d51e4ad2597bf6c464c5b
Haibo Tong
690d51e4ad2597bf6c464c5c
Huaxiu Yao
690d51e4ad2597bf6c464c5d
Hengduo Li
690d51e4ad2597bf6c464c5e
Jiacheng Zhu
claimed_verified
2025-11-10T09:32:05.740Z
653801b167325b6218ddfdc8
https://cdn-avatars.huggingface.co/v1/production/uploads/653801b167325b6218ddfdc8/y3DjEDpcvgnRC30201dFc.jpeg
Jiacheng Zhu
JiachengZhu
user
JiachengZhu
690d51e4ad2597bf6c464c5f
Xian Li
690d51e4ad2597bf6c464c60
Dawn Song
690d51e4ad2597bf6c464c61
Bo Li
690d51e4ad2597bf6c464c62
Jason Weston
690d51e4ad2597bf6c464c63
Dat Huynh
Scaling Agent Learning via Experience Synthesis
While reinforcement learning (RL) can empower large language model (LLM)
agents by enabling self-improvement through interaction, its practical adoption
remains challenging due to costly rollouts, limited task diversity, unreliable
reward signals, and infrastructure complexity, all of which obstruct the
collection of scalable experience data. To address these challenges, we
introduce DreamGym, the first unified framework designed to synthesize diverse
experiences with scalability in mind to enable effective online RL training for
autonomous agents. Rather than relying on expensive real-environment rollouts,
DreamGym distills environment dynamics into a reasoning-based experience model
that derives consistent state transitions and feedback signals through
step-by-step reasoning, enabling scalable agent rollout collection for RL. To
improve the stability and quality of transitions, DreamGym leverages an
experience replay buffer initialized with offline real-world data and
continuously enriched with fresh interactions to actively support agent
training. To improve knowledge acquisition, DreamGym adaptively generates new
tasks that challenge the current agent policy, enabling more effective online
curriculum learning. Experiments across diverse environments and agent
backbones demonstrate that DreamGym substantially improves RL training, both in
fully synthetic settings and in sim-to-real transfer scenarios. On non-RL-ready
tasks like WebArena, DreamGym outperforms all baselines by over 30%. And in
RL-ready but costly settings, it matches GRPO and PPO performance using only
synthetic interactions. When transferring a policy trained purely on synthetic
experiences to real-environment RL, DreamGym yields significant additional
performance gains while requiring far fewer real-world interactions, providing
a scalable warm-start strategy for general-purpose RL.
DreamGym is a unified framework that synthesizes diverse experiences for scalable online RL training, improving agent performance and reducing real-world interactions.
reinforcement learning large language model self-improvement real-environment rollouts experience model state transitions feedback signals experience replay buffer offline real-world data adaptive task generation curriculum learning sim-to-real transfer WebArena GRPO PPO synthetic interactions