EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
Long-term training of large language models (LLMs) requires maintaining stable exploration to prevent the model from collapsing into sub-optimal behaviors. Entropy is crucial in this context, as it controls exploration and helps avoid premature convergence to sub-optimal solutions. However, existing reinforcement learning methods struggle to maintain an appropriate level of entropy, as the training process involves a mix of positive and negative samples, each affecting entropy in different ways across steps. To address this, we propose Entropy stablilization via Proportional-Integral Control (EntroPIC), a novel method that adaptively adjusts the influence of positive and negative samples by dynamically tuning their loss coefficients. This approach stabilizes entropy throughout training, ensuring efficient exploration and steady progress. We provide a comprehensive theoretical analysis for both on-policy and off-policy learning settings, demonstrating that EntroPIC is effective at controlling entropy in large-scale LLM training. Experimental results show that our method successfully maintains desired entropy levels, enabling stable and optimal RL training for LLMs.
691e7f8e3c64d32b03645884
Kai Yang
claimed_verified
2025-11-21T16:49:12.944Z
64930ae54ab21070dc8a42db
/avatars/b341feaf8779b39c225c5b044f29e671.svg
Kai Yang
yangkaiSIGS
user
yangkaiSIGS
691e7f8e3c64d32b03645885
Xin Xu
claimed_verified
2025-12-31T20:58:31.970Z
64e2d169d2af12910d682130
https://cdn-avatars.huggingface.co/v1/production/uploads/64e2d169d2af12910d682130/VG8UdqJCJGc0K4G0P0XQP.jpeg
xuxin
xx18
user
xx18
691e7f8e3c64d32b03645886
Yangkun Chen
691e7f8e3c64d32b03645887
Weijie Liu
691e7f8e3c64d32b03645888
Jiafei Lyu
691e7f8e3c64d32b03645889
Zichuan Lin
claimed_verified
2025-12-04T14:56:19.595Z
687da36e2eaea8261f1323d6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hFH69bJGIDMikEYyClray.png
zichuan lin
zichuan-lin
user
zichuan-lin
691e7f8e3c64d32b0364588a
Deheng Ye
691e7f8e3c64d32b0364588b
Saiyong Yang
EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
Long-term training of large language models (LLMs) requires maintaining stable exploration to prevent the model from collapsing into sub-optimal behaviors. Entropy is crucial in this context, as it controls exploration and helps avoid premature convergence to sub-optimal solutions. However, existing reinforcement learning methods struggle to maintain an appropriate level of entropy, as the training process involves a mix of positive and negative samples, each affecting entropy in different ways across steps. To address this, we propose Entropy stablilization via Proportional-Integral Control (EntroPIC), a novel method that adaptively adjusts the influence of positive and negative samples by dynamically tuning their loss coefficients. This approach stabilizes entropy throughout training, ensuring efficient exploration and steady progress. We provide a comprehensive theoretical analysis for both on-policy and off-policy learning settings, demonstrating that EntroPIC is effective at controlling entropy in large-scale LLM training. Experimental results show that our method successfully maintains desired entropy levels, enabling stable and optimal RL training for LLMs.
EntroPIC, a novel reinforcement learning method, adaptsively tunes loss coefficients to stabilize entropy during long-term training of large language models, ensuring efficient exploration and optimal training.
large language models entropy reinforcement learning positive and negative samples proportional-integral control on-policy learning off-policy learning