Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.
695f2d8a5fa3847525c41f8d
Muxi Diao
claimed_verified
2026-01-08T08:31:56.507Z
6768c97367e4b4606a3c9cec
/avatars/5ddafe7a05828366f66c79072556f370.svg
diaomuxi
diaomuxi
user
diaomuxi
695f2d8a5fa3847525c41f8e
Lele Yang
claimed_verified
2026-01-08T08:31:51.246Z
666a6cf89a3e3ce05a519bcc
/avatars/9e72481deec3bd5c5202e42c32894a32.svg
杨乐乐
ssl-asuka
user
ssl-asuka
695f2d8a5fa3847525c41f8f
Wuxuan Gong
admin_assigned
2026-01-08T08:44:19.470Z
64c0f972d76592ba899c2c9c
/avatars/d6940beb135f99241c6fb2cf0e8ccdbe.svg
GongWuxuan
Wuxuan-Gong
user
Wuxuan-Gong
695f2d8a5fa3847525c41f90
Yutong Zhang
695f2d8a5fa3847525c41f91
Zhonghao Yan
claimed_verified
2026-01-08T08:31:53.904Z
64fbd4e69a62bb2791b3a665
https://cdn-avatars.huggingface.co/v1/production/uploads/64fbd4e69a62bb2791b3a665/ZEMtU8O0z98ryeRCG3l_K.jpeg
Zhonghao Yan
zzzyzh
user
zzzyzh
695f2d8a5fa3847525c41f92
Yufei Han
695f2d8a5fa3847525c41f93
Kongming Liang
admin_assigned
2026-01-08T08:44:38.812Z
67f4a56928cbc4f2f75c008d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/qoX8HT0JsjqW2OQoENSvg.png
Kongming Liang
KongmingLiang
user
KongmingLiang
695f2d8a5fa3847525c41f94
Weiran Xu
695f2d8a5fa3847525c41f95
Zhanyu Ma
Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.
Entropy-Adaptive Fine-Tuning addresses catastrophic forgetting in supervised fine-tuning by using token-level entropy to distinguish uncertainty from knowledge conflict, enabling better preservation of general capabilities.
supervised fine-tuning catastrophic forgetting on-policy reinforcement learning distributional gap Confident Conflicts token-level entropy epistemic uncertainty knowledge conflict gradient updates downstream performance