DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.
699d1f054e37ec6dfa1bc5f1
Zhongwei Wan
699d1f054e37ec6dfa1bc5f2
Yun Shen
699d1f054e37ec6dfa1bc5f3
Zhihao Dou
699d1f054e37ec6dfa1bc5f4
Donghao Zhou
claimed_verified
2026-02-25T17:31:41.803Z
67136093d2e50f1e8c9fad52
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0q49MyGuav8lJ9CIeyLhu.png
Donghao Zhou
donghao-zhou
user
donghao-zhou
699d1f054e37ec6dfa1bc5f5
Yu Zhang
699d1f054e37ec6dfa1bc5f6
Xin Wang
699d1f054e37ec6dfa1bc5f7
Hui Shen
699d1f054e37ec6dfa1bc5f8
Jing Xiong
699d1f054e37ec6dfa1bc5f9
Chaofan Tao
699d1f054e37ec6dfa1bc5fa
Zixuan Zhong
claimed_verified
2026-02-24T09:50:11.851Z
694524196bedf03d6b02cf22
/avatars/733247ed969843cda789e710240477f1.svg
Zixuan ZHONG
Zixuan8642
user
Zixuan8642
699d1f054e37ec6dfa1bc5fb
Peizhou Huang
claimed_verified
2026-03-04T09:24:52.150Z
68060fe82da29557ede62815
https://cdn-avatars.huggingface.co/v1/production/uploads/68060fe82da29557ede62815/CZAMALCB2M3A35Fmid_eM.png
Huang233
Enoch-A
user
Enoch-A
699d1f054e37ec6dfa1bc5fc
Mi Zhang
DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.
DSDR is a reinforcement learning framework that enhances large language model reasoning by promoting diversity at both global and local levels through dual-scale regularization techniques.
reinforcement learning large language models reasoning policy optimization entropy regularization diversity regularization global diversity local diversity token-level entropy trajectory diversity group-based optimization pass@k