GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
69607a225b7998385e63952a
Shih-Yang Liu
claimed_verified
2026-01-09T08:35:01.190Z
62b58c68a1bae3c711c41321
https://cdn-avatars.huggingface.co/v1/production/uploads/62b58c68a1bae3c711c41321/FGQ1ifsPpmRi8P0ElxV5J.png
LIU Shih-yang
sliuau
user
sliuau
69607a225b7998385e63952b
Xin Dong
claimed_verified
2026-01-12T10:36:12.890Z
6092aaac134b77647f624bce
https://cdn-avatars.huggingface.co/v1/production/uploads/6092aaac134b77647f624bce/HGc1cJo7iOFJMKpZWrYJ0.jpeg
Xin Dong
SimonX
user
SimonX
69607a225b7998385e63952c
Ximing Lu
admin_assigned
2026-01-09T08:49:57.401Z
640928bd3461c51cf7378707
/avatars/b29fcb7388b81f8686086352f6321d06.svg
Ximing Lu
Ximing
user
Ximing
69607a225b7998385e63952d
Shizhe Diao
69607a225b7998385e63952e
Peter Belcak
admin_assigned
2026-01-09T15:49:07.360Z
63e8cccddd2c4effdd6283cf
/avatars/b289d4dda0aafd60af3f14e19837b69c.svg
Peter Belcak
pbelcak
user
pbelcak
69607a225b7998385e63952f
Mingjie Liu
69607a225b7998385e639530
Min-Hung Chen
claimed_verified
2026-01-09T08:35:03.130Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
cmhungsteve
user
cmhungsteve
69607a225b7998385e639531
Hongxu Yin
admin_assigned
2026-01-09T15:48:57.052Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
yinhongxu
user
yinhongxu
69607a225b7998385e639532
Yu-Chiang Frank Wang
69607a225b7998385e639533
Kwang-Ting Cheng
69607a225b7998385e639534
Yejin Choi
admin_assigned
2026-01-09T15:48:43.597Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
yejinchoinka
user
yejinchoinka
69607a225b7998385e639535
Jan Kautz
69607a225b7998385e639536
Pavlo Molchanov
admin_assigned
2026-01-09T15:48:21.861Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
pmolchanov
user
pmolchanov
GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
Multi-reward reinforcement learning suffers from reward normalization collapse in GRPO, which GDPO addresses by decoupling reward normalization for improved training stability and performance across reasoning tasks.
Reinforcement learning Group Relative Policy Optimization multi-reward setting policy optimization Group reward-Decoupled Normalization Policy Optimization reward normalization advantage values training stability multi-reward reinforcement learning