Routing Manifold Alignment Improves Generalization of Mixture-of-Experts
  LLMs
Sparse Mixture-of-Experts (MoE) have been widely adopted in recent large
language models since it can efficiently scale up the model capability without
increasing the inference cost. However, evaluations on broad downstream tasks
reveal a consistent suboptimality of the routers in existing MoE LLMs, which
results in a severe performance gap (e.g., 10-20% in accuracy) to the optimal
routing. In this paper, we show that aligning the manifold of routing weights
with that of task embedding can effectively reduce the gap and improve MoE
LLMs' generalization performance. Our method, "Routing Manifold Alignment
(RoMA)", introduces an additional manifold regularization term in the
post-training objective and only requires lightweight finetuning of routers
(with other parameters frozen). Specifically, the regularization encourages the
routing weights of each sample to be close to those of its successful neighbors
(whose routing weights lead to correct answers) in a task embedding space.
Consequently, samples targeting similar tasks will share similar expert choices
across layers. Building such bindings between tasks and experts over different
samples is essential to achieve better generalization. Moreover, RoMA
demonstrates the advantage of unifying the task understanding (by embedding
models) with solution generation (by MoE LLMs). In experiments, we finetune
routers in OLMoE, DeepSeekMoE, and Qwen3-MoE using RoMA. Evaluations on diverse
benchmarks and extensive comparisons with baselines show the substantial
improvement brought by RoMA.
6912af1ca644ba07c499c71a
Zhongyang Li
6912af1ca644ba07c499c71b
Ziyue Li
6912af1ca644ba07c499c71c
Tianyi Zhou
claimed_verified
2025-11-11T19:44:12.162Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
zhoutianyi
user
zhoutianyi
Routing Manifold Alignment Improves Generalization of Mixture-of-Experts
  LLMs
Sparse Mixture-of-Experts (MoE) have been widely adopted in recent large
language models since it can efficiently scale up the model capability without
increasing the inference cost. However, evaluations on broad downstream tasks
reveal a consistent suboptimality of the routers in existing MoE LLMs, which
results in a severe performance gap (e.g., 10-20% in accuracy) to the optimal
routing. In this paper, we show that aligning the manifold of routing weights
with that of task embedding can effectively reduce the gap and improve MoE
LLMs' generalization performance. Our method, "Routing Manifold Alignment
(RoMA)", introduces an additional manifold regularization term in the
post-training objective and only requires lightweight finetuning of routers
(with other parameters frozen). Specifically, the regularization encourages the
routing weights of each sample to be close to those of its successful neighbors
(whose routing weights lead to correct answers) in a task embedding space.
Consequently, samples targeting similar tasks will share similar expert choices
across layers. Building such bindings between tasks and experts over different
samples is essential to achieve better generalization. Moreover, RoMA
demonstrates the advantage of unifying the task understanding (by embedding
models) with solution generation (by MoE LLMs). In experiments, we finetune
routers in OLMoE, DeepSeekMoE, and Qwen3-MoE using RoMA. Evaluations on diverse
benchmarks and extensive comparisons with baselines show the substantial
improvement brought by RoMA.
Aligning routing weights with task embeddings in Sparse Mixture-of-Experts (MoE) models improves generalization and reduces performance gaps in large language models.
Sparse Mixture-of-Experts MoE routing weights task embedding manifold regularization RoMA OLMoE DeepSeekMoE Qwen3-MoE