Stabilizing MoE Reinforcement Learning by Aligning Training and
  Inference Routers
Reinforcement learning (RL) has emerged as a crucial approach for enhancing
the capabilities of large language models. However, in Mixture-of-Experts (MoE)
models, the routing mechanism often introduces instability, even leading to
catastrophic RL training collapse. We analyze the training-inference
consistency of MoE models and identify a notable discrepancy in routing
behaviors between the two phases. Moreover, even under identical conditions,
the routing framework can yield divergent expert selections across repeated
forward passes. To address this foundational inconsistency, we propose Rollout
Routing Replay (R3), a method that records routing distributions from the
inference engine and replays them during training. R3 significantly reduces
training-inference policy KL divergence and mitigates extreme discrepancies
without compromising training speed. Extensive experiments on various settings
confirm that R3 succeeds in stabilizing RL training, preventing collapse and
outperforming methods such as GSPO and TIS. We believe this work can offer a
new solution for stabilizing RL in MoE models.
68ff46536cdff8b857f47319
Wenhan Ma
admin_assigned
2025-10-27T14:46:19.865Z
6686f695840ee769597de318
https://cdn-avatars.huggingface.co/v1/production/uploads/6686f695840ee769597de318/50XZQ80Y2PElI35O0qVFc.jpeg
Wenhan Ma
CuteNPC
user
CuteNPC
68ff46536cdff8b857f4731a
Hailin Zhang
68ff46536cdff8b857f4731b
Liang Zhao
68ff46536cdff8b857f4731c
Yifan Song
68ff46536cdff8b857f4731d
Yudong Wang
68ff46536cdff8b857f4731e
Zhifang Sui
68ff46536cdff8b857f4731f
Fuli Luo
admin_assigned
2025-10-27T14:47:12.114Z
6538815d1bdb3c40db94fbfa
https://cdn-avatars.huggingface.co/v1/production/uploads/6538815d1bdb3c40db94fbfa/id7aSY8JUgKK2agKWLERt.jpeg
Fuli Luo
luofuli
user
luofuli
Stabilizing MoE Reinforcement Learning by Aligning Training and
  Inference Routers
Reinforcement learning (RL) has emerged as a crucial approach for enhancing
the capabilities of large language models. However, in Mixture-of-Experts (MoE)
models, the routing mechanism often introduces instability, even leading to
catastrophic RL training collapse. We analyze the training-inference
consistency of MoE models and identify a notable discrepancy in routing
behaviors between the two phases. Moreover, even under identical conditions,
the routing framework can yield divergent expert selections across repeated
forward passes. To address this foundational inconsistency, we propose Rollout
Routing Replay (R3), a method that records routing distributions from the
inference engine and replays them during training. R3 significantly reduces
training-inference policy KL divergence and mitigates extreme discrepancies
without compromising training speed. Extensive experiments on various settings
confirm that R3 succeeds in stabilizing RL training, preventing collapse and
outperforming methods such as GSPO and TIS. We believe this work can offer a
new solution for stabilizing RL in MoE models.
Rollout Routing Replay (R3) stabilizes reinforcement learning training in Mixture-of-Experts models by reducing discrepancies between training and inference routing behaviors.
Mixture-of-Experts routing mechanism reinforcement learning training-inference consistency Rollout Routing Replay KL divergence GSPO TIS