RAGEN-2: Reasoning Collapse in Agentic RL
RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.
69d70ac3c6ed9968ac5c6967
Zihan Wang
69d70ac3c6ed9968ac5c6968
Chi Gui
claimed_verified
2026-04-10T07:45:14.555Z
6454dbc0e4952d1c6cb34f76
https://cdn-avatars.huggingface.co/v1/production/uploads/6454dbc0e4952d1c6cb34f76/oiV2Zvg8L2pNd2u4TMrdx.jpeg
Chi Gui
Penguin-N
user
Penguin-N
69d70ac3c6ed9968ac5c6969
Xing Jin
claimed_verified
2026-04-09T08:06:51.757Z
67ef78b4efcb0a2fbfca0e41
/avatars/29b5391a807a2991aa0798cff3e8c2c0.svg
Xing
DmsJin
user
DmsJin
69d70ac3c6ed9968ac5c696a
Qineng Wang
69d70ac3c6ed9968ac5c696b
Licheng Liu
claimed_verified
2026-04-09T08:06:53.895Z
661caf2f6158c41091b3954e
https://cdn-avatars.huggingface.co/v1/production/uploads/661caf2f6158c41091b3954e/zrAqbiL4u1l6_Loea00_k.jpeg
Licheng Liu
LichengLiu03
user
LichengLiu03
69d70ac3c6ed9968ac5c696c
Kangrui Wang
69d70ac3c6ed9968ac5c696d
Shiqi Chen
69d70ac3c6ed9968ac5c696e
Linjie Li
69d70ac3c6ed9968ac5c696f
Zhengyuan Yang
69d70ac3c6ed9968ac5c6970
Pingyue Zhang
69d70ac3c6ed9968ac5c6971
Yiping Lu
69d70ac3c6ed9968ac5c6972
Jiajun Wu
69d70ac3c6ed9968ac5c6973
Li Fei-Fei
69d70ac3c6ed9968ac5c6974
Lijuan Wang
69d70ac3c6ed9968ac5c6975
Yejin Choi
69d70ac3c6ed9968ac5c6976
Manling Li
claimed_verified
2026-04-13T07:34:18.949Z
6746a140f2ca2162e3bcfe2b
/avatars/d9d8cfb5f112e6ed7f6152fc230135d3.svg
Manling Li
ManlingLi
user
ManlingLi
RAGEN-2: Reasoning Collapse in Agentic RL
RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.
Research identifies template collapse in multi-turn LLM agents as a hidden failure mode undetectable by entropy, proposing mutual information proxies and SNR-aware filtering to improve reasoning quality and task performance.
entropy mutual information template collapse signal-to-noise ratio reward variance reasoning quality input dependence task performance reinforcement learning large language models