Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
The primary obstacle for applying reinforcement learning (RL) to real-world robotics is the design of effective reward functions. While recently learning-based Process Reward Models (PRMs) are a promising direction, they are often hindered by two fundamental limitations: their reward models lack step-aware understanding and rely on single-view perception, leading to unreliable assessments of fine-grained manipulation progress; and their reward shaping procedures are theoretically unsound, often inducing a semantic trap that misguides policy optimization. To address these, we introduce Dopamine-Reward, a novel reward modeling method for learning a general-purpose, step-aware process reward model from multi-view inputs. At its core is our General Reward Model (GRM), trained on a vast 3,400+ hour dataset, which leverages Step-wise Reward Discretization for structural understanding and Multi-Perspective Reward Fusion to overcome perceptual limitations. Building upon Dopamine-Reward, we propose Dopamine-RL, a robust policy learning framework that employs a theoretically-sound Policy-Invariant Reward Shaping method, which enables the agent to leverage dense rewards for efficient self-improvement without altering the optimal policy, thereby fundamentally avoiding the semantic trap. Extensive experiments across diverse simulated and real-world tasks validate our approach. GRM achieves state-of-the-art accuracy in reward assessment, and Dopamine-RL built on GRM significantly improves policy learning efficiency. For instance, after GRM is adapted to a new task in a one-shot manner from a single expert trajectory, the resulting reward model enables Dopamine-RL to improve the policy from near-zero to 95% success with only 150 online rollouts (approximately 1 hour of real robot interaction), while retaining strong generalization across tasks. Project website: https://robo-dopamine.github.io
6953cdc209c8c0a5381814e1
Huajie Tan
claimed_verified
2025-12-31T20:55:02.091Z
66b5dc0b854ad316cf835ab4
https://cdn-avatars.huggingface.co/v1/production/uploads/66b5dc0b854ad316cf835ab4/8gOWw81rV5la7mzjw_qRv.jpeg
tanhuajie2001
tanhuajie2001
user
tanhuajie2001
6953cdc209c8c0a5381814e2
Sixiang Chen
6953cdc209c8c0a5381814e3
Yijie Xu
claimed_verified
2025-12-31T20:54:58.653Z
6672dbb6bf11c2d4404ce64a
/avatars/655f3c4bf44850be9f3e73be3a64f353.svg
Yijie Xu
YijieXuJoey
user
YijieXuJoey
6953cdc209c8c0a5381814e4
Zixiao Wang
claimed_verified
2025-12-31T20:54:55.887Z
66cb02bd315af068a95b4cd1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ES7lfvsN-wF_r2fFRNEPx.png
zixiao_bios
zixiao-bios
user
zixiao-bios
6953cdc209c8c0a5381814e5
Yuheng Ji
6953cdc209c8c0a5381814e6
Cheng Chi
6953cdc209c8c0a5381814e7
Yaoxu Lyu
6953cdc209c8c0a5381814e8
Zhongxia Zhao
6953cdc209c8c0a5381814e9
Xiansheng Chen
6953cdc209c8c0a5381814ea
Peterson Co
6953cdc209c8c0a5381814eb
Shaoxuan Xie
6953cdc209c8c0a5381814ec
Guocai Yao
6953cdc209c8c0a5381814ed
Pengwei Wang
6953cdc209c8c0a5381814ee
Zhongyuan Wang
6953cdc209c8c0a5381814ef
Shanghang Zhang
Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
The primary obstacle for applying reinforcement learning (RL) to real-world robotics is the design of effective reward functions. While recently learning-based Process Reward Models (PRMs) are a promising direction, they are often hindered by two fundamental limitations: their reward models lack step-aware understanding and rely on single-view perception, leading to unreliable assessments of fine-grained manipulation progress; and their reward shaping procedures are theoretically unsound, often inducing a semantic trap that misguides policy optimization. To address these, we introduce Dopamine-Reward, a novel reward modeling method for learning a general-purpose, step-aware process reward model from multi-view inputs. At its core is our General Reward Model (GRM), trained on a vast 3,400+ hour dataset, which leverages Step-wise Reward Discretization for structural understanding and Multi-Perspective Reward Fusion to overcome perceptual limitations. Building upon Dopamine-Reward, we propose Dopamine-RL, a robust policy learning framework that employs a theoretically-sound Policy-Invariant Reward Shaping method, which enables the agent to leverage dense rewards for efficient self-improvement without altering the optimal policy, thereby fundamentally avoiding the semantic trap. Extensive experiments across diverse simulated and real-world tasks validate our approach. GRM achieves state-of-the-art accuracy in reward assessment, and Dopamine-RL built on GRM significantly improves policy learning efficiency. For instance, after GRM is adapted to a new task in a one-shot manner from a single expert trajectory, the resulting reward model enables Dopamine-RL to improve the policy from near-zero to 95% success with only 150 online rollouts (approximately 1 hour of real robot interaction), while retaining strong generalization across tasks. Project website: https://robo-dopamine.github.io
A novel reward modeling approach called Dopamine-Reward addresses limitations in reinforcement learning for robotics by introducing a step-aware process reward model and theoretically sound reward shaping to improve policy learning efficiency and generalization.
Process Reward Models step-aware understanding multi-view perception reward modeling General Reward Model Step-wise Reward Discretization Multi-Perspective Reward Fusion Policy-Invariant Reward Shaping policy learning reinforcement learning