From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
We introduce Distribution Contractive Reinforcement Learning (DICE-RL), a framework that uses reinforcement learning (RL) as a "distribution contraction" operator to refine pretrained generative robot policies. DICE-RL turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback. We pretrain a diffusion- or flow-based policy for broad behavioral coverage, then finetune it with a stable, sample-efficient residual off-policy RL framework that combines selective behavior regularization with value-guided action selection. Extensive experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency. It enables mastery of complex long-horizon manipulation skills directly from high-dimensional pixel inputs, both in simulation and on a real robot. Project website: https://zhanyisun.github.io/dice.rl.2026/.
69bb8347ba2b62a5fb0b0d32
Zhanyi Sun
claimed_verified
2026-03-19T12:57:02.324Z
69776bf9ddc564e5acf5e4ea
https://cdn-avatars.huggingface.co/v1/production/uploads/69776bf9ddc564e5acf5e4ea/e7WOK2gP1XqPKGxnwmge_.jpeg
Zhanyi Sun
wintermelontree
user
wintermelontree
69bb8347ba2b62a5fb0b0d33
Shuran Song
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
We introduce Distribution Contractive Reinforcement Learning (DICE-RL), a framework that uses reinforcement learning (RL) as a "distribution contraction" operator to refine pretrained generative robot policies. DICE-RL turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback. We pretrain a diffusion- or flow-based policy for broad behavioral coverage, then finetune it with a stable, sample-efficient residual off-policy RL framework that combines selective behavior regularization with value-guided action selection. Extensive experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency. It enables mastery of complex long-horizon manipulation skills directly from high-dimensional pixel inputs, both in simulation and on a real robot. Project website: https://zhanyisun.github.io/dice.rl.2026/.
DICE-RL enhances pretrained generative robot policies through reinforcement learning distribution contraction, achieving complex manipulation skills from pixel inputs with improved stability and sample efficiency.
reinforcement learning distribution contraction pretrained generative robot policies diffusion-based policy flow-based policy residual off-policy RL selective behavior regularization value-guided action selection long-horizon manipulation skills pixel inputs