STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens
Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often experience late-stage performance collapse, leading to degraded reasoning quality and unstable training. We derive that the magnitude of token-wise policy gradients in RL is negatively correlated with token probability and local policy entropy. Building on this result, we prove that training instability is driven by a tiny fraction of tokens, approximately 0.01\%, which we term spurious tokens. When such tokens appear in correct responses, they contribute little to the reasoning outcome but inherit the full sequence-level reward, leading to abnormally amplified gradient updates. Motivated by this observation, we propose Spurious-Token-Aware Policy Optimization (STAPO) for large-scale model refining, which selectively masks such updates and renormalizes the loss over valid tokens. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 7.13\% over GRPO, 20-Entropy and JustRL.
69952cdc8d17d1ee8c10ec9c
Shiqi Liu
admin_assigned
2026-02-18T13:42:07.826Z
68d42366298c4a04b90bc575
/avatars/51df491a1e6cdcc5887444620692a502.svg
Shiqi Liu
ShiqiLiu
user
ShiqiLiu
69952cdc8d17d1ee8c10ec9d
Zeyu He
69952cdc8d17d1ee8c10ec9e
Guojian Zhan
69952cdc8d17d1ee8c10ec9f
Letian Tao
69952cdc8d17d1ee8c10eca0
Zhilong Zheng
admin_assigned
2026-02-18T13:42:28.293Z
68ecc8477b9cd18b1020719e
/avatars/ebe459c2b2ecf124cba04590d26b6123.svg
Zhilong Zheng
zzzzl-h
user
zzzzl-h
69952cdc8d17d1ee8c10eca1
Jiang Wu
69952cdc8d17d1ee8c10eca2
Yinuo Wang
69952cdc8d17d1ee8c10eca3
Yang Guan
69952cdc8d17d1ee8c10eca4
Kehua Sheng
admin_assigned
2026-02-18T13:42:39.054Z
64b0fc8f4db8a01408ce1042
/avatars/8eb265c114444065ddff6e8f06126666.svg
Kehua Sheng
KehuaSheng
user
KehuaSheng
69952cdc8d17d1ee8c10eca5
Bo Zhang
69952cdc8d17d1ee8c10eca6
Keqiang Li
admin_assigned
2026-02-18T13:42:45.671Z
65bb68849170f742fd10aff5
/avatars/42a7b3ab2132132d409d39e67f152dec.svg
Keqiang Li
UestcJay
user
UestcJay
69952cdc8d17d1ee8c10eca7
Jingliang Duan
69952cdc8d17d1ee8c10eca8
Shengbo Eben Li
STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens
Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often experience late-stage performance collapse, leading to degraded reasoning quality and unstable training. We derive that the magnitude of token-wise policy gradients in RL is negatively correlated with token probability and local policy entropy. Building on this result, we prove that training instability is driven by a tiny fraction of tokens, approximately 0.01\%, which we term spurious tokens. When such tokens appear in correct responses, they contribute little to the reasoning outcome but inherit the full sequence-level reward, leading to abnormally amplified gradient updates. Motivated by this observation, we propose Spurious-Token-Aware Policy Optimization (STAPO) for large-scale model refining, which selectively masks such updates and renormalizes the loss over valid tokens. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 7.13\% over GRPO, 20-Entropy and JustRL.
Research identifies spurious tokens as the cause of training instability in reinforcement learning fine-tuning of large language models and proposes a solution that selectively masks problematic gradient updates to improve reasoning performance.
reinforcement learning policy gradients token probability policy entropy training instability spurious tokens policy optimization gradient updates mathematical reasoning benchmarks Qwen models GRPO Entropy JustRL