IIB-LPO: Latent Policy Optimization via Iterative Information Bottleneck
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Model (LLM) reasoning have been hindered by a persistent challenge: exploration collapse. The semantic homogeneity of random rollouts often traps models in narrow, over-optimized behaviors. While existing methods leverage policy entropy to encourage exploration, they face inherent limitations. Global entropy regularization is susceptible to reward hacking, which can induce meaningless verbosity, whereas local token-selective updates struggle with the strong inductive bias of pre-trained models. To address this, we propose Latent Policy Optimization via Iterative Information Bottleneck (IIB-LPO), a novel approach that shifts exploration from statistical perturbation of token distributions to topological branching of reasoning trajectories. IIB-LPO triggers latent branching at high-entropy states to diversify reasoning paths and employs the Information Bottleneck principle both as a trajectory filter and a self-reward mechanism, ensuring concise and informative exploration. Empirical results across four mathematical reasoning benchmarks demonstrate that IIB-LPO achieves state-of-the-art performance, surpassing prior methods by margins of up to 5.3% in accuracy and 7.4% in diversity metrics.
69646166138cc47cbd765271
Huilin Deng
69646166138cc47cbd765272
Hongchen Luo
69646166138cc47cbd765273
Yue Zhu
69646166138cc47cbd765274
Long Li
69646166138cc47cbd765275
Zhuoyue Chen
69646166138cc47cbd765276
Xinghao Zhao
69646166138cc47cbd765277
Ming Li
69646166138cc47cbd765278
Jihai Zhang
69646166138cc47cbd765279
Mengchang Wang
69646166138cc47cbd76527a
Yang Cao
69646166138cc47cbd76527b
Yu Kang
IIB-LPO: Latent Policy Optimization via Iterative Information Bottleneck
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Model (LLM) reasoning have been hindered by a persistent challenge: exploration collapse. The semantic homogeneity of random rollouts often traps models in narrow, over-optimized behaviors. While existing methods leverage policy entropy to encourage exploration, they face inherent limitations. Global entropy regularization is susceptible to reward hacking, which can induce meaningless verbosity, whereas local token-selective updates struggle with the strong inductive bias of pre-trained models. To address this, we propose Latent Policy Optimization via Iterative Information Bottleneck (IIB-LPO), a novel approach that shifts exploration from statistical perturbation of token distributions to topological branching of reasoning trajectories. IIB-LPO triggers latent branching at high-entropy states to diversify reasoning paths and employs the Information Bottleneck principle both as a trajectory filter and a self-reward mechanism, ensuring concise and informative exploration. Empirical results across four mathematical reasoning benchmarks demonstrate that IIB-LPO achieves state-of-the-art performance, surpassing prior methods by margins of up to 5.3% in accuracy and 7.4% in diversity metrics.
Latent Policy Optimization via Iterative Information Bottleneck addresses exploration collapse in LLM reasoning by enabling topological branching of reasoning trajectories through information bottleneck principles.
Reinforcement Learning with Verifiable Rewards large language models exploration collapse policy entropy reward hacking latent policy optimization iterative information bottleneck topological branching reasoning trajectories information bottleneck principle mathematical reasoning benchmarks