GoRL: An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies
Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize are often too simple to represent the multimodal action distributions needed for complex control. Gaussian policies provide tractable likelihoods and smooth gradients, but their unimodal form limits expressiveness. Conversely, generative policies based on diffusion or flow matching can model rich multimodal behaviors; however, in online RL, they are frequently unstable due to intractable likelihoods and noisy gradients propagating through deep sampling chains. We address this tension with a key structural principle: decoupling optimization from generation. Building on this insight, we introduce GoRL (Generative Online Reinforcement Learning), a framework that optimizes a tractable latent policy while utilizing a conditional generative decoder to synthesize actions. A two-timescale update schedule enables the latent policy to learn stably while the decoder steadily increases expressiveness, without requiring tractable action likelihoods. Across a range of continuous-control tasks, GoRL consistently outperforms both Gaussian policies and recent generative-policy baselines. Notably, on the HopperStand task, it reaches a normalized return above 870, more than 3 times that of the strongest baseline. These results demonstrate that separating optimization from generation provides a practical path to policies that are both stable and highly expressive.
692fc6fa26742347f61dad06
Chubin Zhang
692fc6fa26742347f61dad07
Zhenglin Wan
claimed_verified
2025-12-03T09:15:09.779Z
6594db22e9d0cd61bc7b7de8
/avatars/3ef34c3d3db195b90e438a77a9efed37.svg
Zhenglin Wan
Carlos133386
user
Carlos133386
692fc6fa26742347f61dad08
Feng Chen
692fc6fa26742347f61dad09
Xingrui Yu
692fc6fa26742347f61dad0a
Ivor Tsang
692fc6fa26742347f61dad0b
Bo An
GoRL: An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies
Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize are often too simple to represent the multimodal action distributions needed for complex control. Gaussian policies provide tractable likelihoods and smooth gradients, but their unimodal form limits expressiveness. Conversely, generative policies based on diffusion or flow matching can model rich multimodal behaviors; however, in online RL, they are frequently unstable due to intractable likelihoods and noisy gradients propagating through deep sampling chains. We address this tension with a key structural principle: decoupling optimization from generation. Building on this insight, we introduce GoRL (Generative Online Reinforcement Learning), a framework that optimizes a tractable latent policy while utilizing a conditional generative decoder to synthesize actions. A two-timescale update schedule enables the latent policy to learn stably while the decoder steadily increases expressiveness, without requiring tractable action likelihoods. Across a range of continuous-control tasks, GoRL consistently outperforms both Gaussian policies and recent generative-policy baselines. Notably, on the HopperStand task, it reaches a normalized return above 870, more than 3 times that of the strongest baseline. These results demonstrate that separating optimization from generation provides a practical path to policies that are both stable and highly expressive.
GoRL separates optimization and generation to achieve stable and expressive policies in reinforcement learning, outperforming Gaussian and generative-policy baselines.
Gaussian policies diffusion flow matching generative policies latent policy conditional generative decoder two-timescale update schedule continuous-control tasks HopperStand task normalized return