Critique-RL: Training Language Models for Critiquing through Two-Stage
  Reinforcement Learning
Training critiquing language models to assess and provide feedback on model
outputs is a promising way to improve LLMs for complex reasoning tasks.
However, existing approaches typically rely on stronger supervisors for
annotating critique data. To address this, we propose Critique-RL, an online RL
approach for developing critiquing language models without stronger
supervision. Our approach operates on a two-player paradigm: the actor
generates a response, the critic provides feedback, and the actor refines the
response accordingly. We first reveal that relying solely on indirect reward
signals from the actor's outputs for RL optimization often leads to
unsatisfactory critics: while their helpfulness (i.e., providing constructive
feedback) improves, the discriminability (i.e., determining whether a response
is high-quality or not) remains poor, resulting in marginal performance gains.
To overcome this, Critique-RL adopts a two-stage optimization strategy. In
stage I, it reinforces the discriminability of the critic with direct
rule-based reward signals; in stage II, it introduces indirect rewards based on
actor refinement to improve the critic's helpfulness, while maintaining its
discriminability via appropriate regularization. Extensive experiments across
various tasks and models show that Critique-RL delivers substantial performance
improvements. For example, it achieves a 9.02% gain on in-domain tasks and a
5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential.
69017878646208eac0d1f3aa
Zhiheng Xi
69017878646208eac0d1f3ab
Jixuan Huang
69017878646208eac0d1f3ac
Xin Guo
69017878646208eac0d1f3ad
Boyang Hong
69017878646208eac0d1f3ae
Dingwen Yang
69017878646208eac0d1f3af
Xiaoran Fan
69017878646208eac0d1f3b0
Shuo Li
69017878646208eac0d1f3b1
Zehui Chen
claimed_verified
2025-12-01T09:11:41.631Z
64892d31cbda0d1cdb956897
/avatars/3cdafe03a8295124636347d15a099aaf.svg
Zehui Chen
lovesnowbest
user
lovesnowbest
69017878646208eac0d1f3b2
Junjie Ye
claimed_verified
2025-10-29T12:45:14.912Z
66384be673c2c55f2ded89fa
/avatars/1d8721074f0f51fab405f81474f2035f.svg
Junjie Ye
Junjie-Ye
user
Junjie-Ye
69017878646208eac0d1f3b3
Siyu Yuan
69017878646208eac0d1f3b4
Zhengyin Du
69017878646208eac0d1f3b5
Xuesong Yao
69017878646208eac0d1f3b6
Yufei Xu
69017878646208eac0d1f3b7
Jiecao Chen
69017878646208eac0d1f3b8
Rui Zheng
69017878646208eac0d1f3b9
Tao Gui
69017878646208eac0d1f3ba
Qi Zhang
69017878646208eac0d1f3bb
Xuanjing Huang
Critique-RL: Training Language Models for Critiquing through Two-Stage
  Reinforcement Learning
Training critiquing language models to assess and provide feedback on model
outputs is a promising way to improve LLMs for complex reasoning tasks.
However, existing approaches typically rely on stronger supervisors for
annotating critique data. To address this, we propose Critique-RL, an online RL
approach for developing critiquing language models without stronger
supervision. Our approach operates on a two-player paradigm: the actor
generates a response, the critic provides feedback, and the actor refines the
response accordingly. We first reveal that relying solely on indirect reward
signals from the actor's outputs for RL optimization often leads to
unsatisfactory critics: while their helpfulness (i.e., providing constructive
feedback) improves, the discriminability (i.e., determining whether a response
is high-quality or not) remains poor, resulting in marginal performance gains.
To overcome this, Critique-RL adopts a two-stage optimization strategy. In
stage I, it reinforces the discriminability of the critic with direct
rule-based reward signals; in stage II, it introduces indirect rewards based on
actor refinement to improve the critic's helpfulness, while maintaining its
discriminability via appropriate regularization. Extensive experiments across
various tasks and models show that Critique-RL delivers substantial performance
improvements. For example, it achieves a 9.02% gain on in-domain tasks and a
5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential.
Critique-RL is an online reinforcement learning approach for developing critiquing language models without strong supervision, using a two-stage optimization strategy to improve both the critic's discriminability and helpfulness.
reinforcement learning RL actor critic indirect reward signals direct rule-based reward signals discriminability helpfulness Qwen2.5-7B