Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies
Existing reinforcement learning (RL) approaches treat large language models (LLMs) as a single unified policy, overlooking their internal mechanisms. Understanding how policy evolves across layers and modules is therefore crucial for enabling more targeted optimization and raveling out complex reasoning mechanisms. In this paper, we decompose the language model policy by leveraging the intrinsic split of the Transformer residual stream and the equivalence between the composition of hidden states with the unembedding matrix and the resulting samplable policy. This decomposition reveals Internal Layer Policies, corresponding to contributions from individual layers, and Internal Modular Policies, which align with the self-attention and feed-forward network (FFN) components within each layer. By analyzing the entropy of internal policy, we find that: (a) Early layers keep high entropy for exploration, top layers converge to near-zero entropy for refinement, with convergence patterns varying across model series. (b) LLama's prediction space rapidly converges in the final layer, whereas Qwen-series models, especially Qwen3, exhibit a more human-like, progressively structured reasoning pattern. Motivated by these findings, we propose Bottom-up Policy Optimization (BuPO), a novel RL paradigm that directly optimizes the internal layer policy during early training. By aligning training objective at lower layer, BuPO reconstructs foundational reasoning capabilities and achieves superior performance. Extensive experiments on complex reasoning benchmarks demonstrates the effectiveness of our method. Our code is available at https://github.com/Trae1ounG/BuPO.
694ac3ad746a34b55dd53b6c
Yuqiao Tan
claimed_verified
2025-12-25T20:46:46.569Z
64feb928c3329fa5933ebf9a
https://cdn-avatars.huggingface.co/v1/production/uploads/64feb928c3329fa5933ebf9a/DOyWiqah1dxjafj6TTGv5.jpeg
TanYuQiao
Trae1ounG
user
Trae1ounG
694ac3ad746a34b55dd53b6d
Minzheng Wang
claimed_verified
2025-12-25T20:46:49.676Z
64bcc373ef8c0e42bf16acc5
/avatars/873308203d28115ae1a9e4d0e26508f4.svg
mz.w
iiiiwis
user
iiiiwis
694ac3ad746a34b55dd53b6e
Shizhu He
694ac3ad746a34b55dd53b6f
Huanxuan Liao
694ac3ad746a34b55dd53b70
Chengfeng Zhao
694ac3ad746a34b55dd53b71
Qiunan Lu
694ac3ad746a34b55dd53b72
Tian Liang
claimed_verified
2026-02-10T09:28:01.591Z
643645baaa4211ef553f613c
https://cdn-avatars.huggingface.co/v1/production/uploads/643645baaa4211ef553f613c/svUVeTqwLf5ZurprdTOUC.jpeg
TimLeung
skytliang
user
skytliang
694ac3ad746a34b55dd53b73
Jun Zhao
694ac3ad746a34b55dd53b74
Kang Liu
Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies
Existing reinforcement learning (RL) approaches treat large language models (LLMs) as a single unified policy, overlooking their internal mechanisms. Understanding how policy evolves across layers and modules is therefore crucial for enabling more targeted optimization and raveling out complex reasoning mechanisms. In this paper, we decompose the language model policy by leveraging the intrinsic split of the Transformer residual stream and the equivalence between the composition of hidden states with the unembedding matrix and the resulting samplable policy. This decomposition reveals Internal Layer Policies, corresponding to contributions from individual layers, and Internal Modular Policies, which align with the self-attention and feed-forward network (FFN) components within each layer. By analyzing the entropy of internal policy, we find that: (a) Early layers keep high entropy for exploration, top layers converge to near-zero entropy for refinement, with convergence patterns varying across model series. (b) LLama's prediction space rapidly converges in the final layer, whereas Qwen-series models, especially Qwen3, exhibit a more human-like, progressively structured reasoning pattern. Motivated by these findings, we propose Bottom-up Policy Optimization (BuPO), a novel RL paradigm that directly optimizes the internal layer policy during early training. By aligning training objective at lower layer, BuPO reconstructs foundational reasoning capabilities and achieves superior performance. Extensive experiments on complex reasoning benchmarks demonstrates the effectiveness of our method. Our code is available at https://github.com/Trae1ounG/BuPO.
The paper decomposes the policy of large language models into internal layer and modular policies, revealing distinct reasoning patterns across layers and proposing Bottom-up Policy Optimization to enhance performance on complex reasoning tasks.
reinforcement learning large language models Transformer residual stream unembedding matrix Internal Layer Policies Internal Modular Policies self-attention feed-forward network entropy Bottom-up Policy Optimization