Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models
Diffusion language models (DLMs) enable parallel, non-autoregressive text generation, yet existing DLM mixture-of-experts (MoE) models inherit token-choice (TC) routing from autoregressive systems, leading to load imbalance and rigid computation allocation. We show that expert-choice (EC) routing is a better fit for DLMs: it provides deterministic load balancing by design, yielding higher throughput and faster convergence than TC. Building on the property that EC capacity is externally controllable, we introduce timestep-dependent expert capacity, which varies expert allocation according to the denoising step. We find that allocating more capacity to low-mask-ratio steps consistently achieves the best performance under matched FLOPs, and provide a mechanistic explanation: tokens in low-mask-ratio contexts exhibit an order-of-magnitude higher learning efficiency, so concentrating compute on these steps yields the largest marginal return. Finally, we show that existing pretrained TC DLMs can be retrofitted to EC by replacing only the router, achieving faster convergence and improved accuracy across diverse downstream tasks. Together, these results establish EC routing as a superior paradigm for DLM MoE models and demonstrate that computation in DLMs can be treated as an adaptive policy rather than a fixed architectural constant. Code is available at https://github.com/zhangshuibai/EC-DLM.
69d67801c6ed9968ac5c674b
Shuibai Zhang
claimed_verified
2026-04-08T16:33:55.988Z
666ff0e9251af54f960e9870
/avatars/c6b727821e29f9d40afdf0ef6935cd0b.svg
Shuibai Zhang
Shuibai12138
user
Shuibai12138
69d67801c6ed9968ac5c674c
Caspian Zhuang
claimed_verified
2026-04-13T07:34:42.630Z
6575a625b951d40e7a4d8685
https://cdn-avatars.huggingface.co/v1/production/uploads/6575a625b951d40e7a4d8685/mcnyqU--r-vi11aXN02cZ.jpeg
Xialie Zhuang
Lie24
user
Lie24
69d67801c6ed9968ac5c674d
Chihan Cui
69d67801c6ed9968ac5c674e
Zhihan Yang
69d67801c6ed9968ac5c674f
Fred Zhangzhi Peng
69d67801c6ed9968ac5c6750
Yanxin Zhang
69d67801c6ed9968ac5c6751
Haoyue Bai
69d67801c6ed9968ac5c6752
Zack Jia
69d67801c6ed9968ac5c6753
Yang Zhou
69d67801c6ed9968ac5c6754
Guanhua Chen
69d67801c6ed9968ac5c6755
Ming Liu
Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models
Diffusion language models (DLMs) enable parallel, non-autoregressive text generation, yet existing DLM mixture-of-experts (MoE) models inherit token-choice (TC) routing from autoregressive systems, leading to load imbalance and rigid computation allocation. We show that expert-choice (EC) routing is a better fit for DLMs: it provides deterministic load balancing by design, yielding higher throughput and faster convergence than TC. Building on the property that EC capacity is externally controllable, we introduce timestep-dependent expert capacity, which varies expert allocation according to the denoising step. We find that allocating more capacity to low-mask-ratio steps consistently achieves the best performance under matched FLOPs, and provide a mechanistic explanation: tokens in low-mask-ratio contexts exhibit an order-of-magnitude higher learning efficiency, so concentrating compute on these steps yields the largest marginal return. Finally, we show that existing pretrained TC DLMs can be retrofitted to EC by replacing only the router, achieving faster convergence and improved accuracy across diverse downstream tasks. Together, these results establish EC routing as a superior paradigm for DLM MoE models and demonstrate that computation in DLMs can be treated as an adaptive policy rather than a fixed architectural constant. Code is available at https://github.com/zhangshuibai/EC-DLM.
Expert-choice routing improves diffusion language model mixture-of-experts by providing deterministic load balancing and adaptive computation allocation based on denoising steps.
diffusion language models mixture-of-experts token-choice routing expert-choice routing load balancing denoising step expert capacity computational adaptation