No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning
Critique-guided reinforcement learning (RL) has emerged as a powerful paradigm for training LLM agents by augmenting sparse outcome rewards with natural-language feedback. However, current methods often rely on static or offline critic models, which fail to adapt as the policy evolves. In on-policy RL, the agent's error patterns shift over time, causing stationary critics to become stale and providing feedback of diminishing utility. To address this, we introduce ECHO (Evolving Critic for Hindsight-Guided Optimization)}, a framework that jointly optimizes the policy and critic through a synchronized co-evolutionary loop. ECHO utilizes a cascaded rollout mechanism where the critic generates multiple diagnoses for an initial trajectory, followed by policy refinement to enable group-structured advantage estimation. We address the challenge of learning plateaus via a saturation-aware gain shaping objective, which rewards the critic for inducing incremental improvements in high-performing trajectories. By employing dual-track GRPO updates, ECHO ensures the critic's feedback stays synchronized with the evolving policy. Experimental results show that ECHO yields more stable training and higher long-horizon task success across open-world environments.
696918718854afb90e2fc9e2
Zhicong Li
696918718854afb90e2fc9e3
Lingjie Jiang
696918718854afb90e2fc9e4
Yulan Hu
696918718854afb90e2fc9e5
Xingchen Zeng
696918718854afb90e2fc9e6
Yixia Li
696918718854afb90e2fc9e7
Xiangwen Zhang
696918718854afb90e2fc9e8
Guanhua Chen
696918718854afb90e2fc9e9
Zheng Pan
696918718854afb90e2fc9ea
Xin Li
696918718854afb90e2fc9eb
Yong Liu
No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning
Critique-guided reinforcement learning (RL) has emerged as a powerful paradigm for training LLM agents by augmenting sparse outcome rewards with natural-language feedback. However, current methods often rely on static or offline critic models, which fail to adapt as the policy evolves. In on-policy RL, the agent's error patterns shift over time, causing stationary critics to become stale and providing feedback of diminishing utility. To address this, we introduce ECHO (Evolving Critic for Hindsight-Guided Optimization)}, a framework that jointly optimizes the policy and critic through a synchronized co-evolutionary loop. ECHO utilizes a cascaded rollout mechanism where the critic generates multiple diagnoses for an initial trajectory, followed by policy refinement to enable group-structured advantage estimation. We address the challenge of learning plateaus via a saturation-aware gain shaping objective, which rewards the critic for inducing incremental improvements in high-performing trajectories. By employing dual-track GRPO updates, ECHO ensures the critic's feedback stays synchronized with the evolving policy. Experimental results show that ECHO yields more stable training and higher long-horizon task success across open-world environments.
ECHO is a reinforcement learning framework that jointly optimizes policy and critic through co-evolutionary loops, addressing staleness issues in critique-guided training of language model agents.
reinforcement learning LLM agents sparse outcome rewards natural-language feedback stationary critics on-policy RL policy evolution critic adaptation co-evolutionary loop cascaded rollout mechanism group-structured advantage estimation saturation-aware gain shaping dual-track GRPO updates