Routing Matters in MoE: Scaling Diffusion Transformers with Explicit
  Routing Guidance
Mixture-of-Experts (MoE) has emerged as a powerful paradigm for scaling model
capacity while preserving computational efficiency. Despite its notable success
in large language models (LLMs), existing attempts to apply MoE to Diffusion
Transformers (DiTs) have yielded limited gains. We attribute this gap to
fundamental differences between language and visual tokens. Language tokens are
semantically dense with pronounced inter-token variation, while visual tokens
exhibit spatial redundancy and functional heterogeneity, hindering expert
specialization in vision MoE. To this end, we present ProMoE, an MoE framework
featuring a two-step router with explicit routing guidance that promotes expert
specialization. Specifically, this guidance encourages the router to partition
image tokens into conditional and unconditional sets via conditional routing
according to their functional roles, and refine the assignments of conditional
image tokens through prototypical routing with learnable prototypes based on
semantic content. Moreover, the similarity-based expert allocation in latent
space enabled by prototypical routing offers a natural mechanism for
incorporating explicit semantic guidance, and we validate that such guidance is
crucial for vision MoE. Building on this, we propose a routing contrastive loss
that explicitly enhances the prototypical routing process, promoting
intra-expert coherence and inter-expert diversity. Extensive experiments on
ImageNet benchmark demonstrate that ProMoE surpasses state-of-the-art methods
under both Rectified Flow and DDPM training objectives. Code and models will be
made publicly available.
6901814d646208eac0d1f468
Yujie Wei
6901814d646208eac0d1f469
Shiwei Zhang
6901814d646208eac0d1f46a
Hangjie Yuan
6901814d646208eac0d1f46b
Yujin Han
6901814d646208eac0d1f46c
Zhekai Chen
6901814d646208eac0d1f46d
Jiayu Wang
6901814d646208eac0d1f46e
Difan Zou
6901814d646208eac0d1f46f
Xihui Liu
6901814d646208eac0d1f470
Yingya Zhang
6901814d646208eac0d1f471
Yu Liu
6901814d646208eac0d1f472
Hongming Shan
Routing Matters in MoE: Scaling Diffusion Transformers with Explicit
  Routing Guidance
Mixture-of-Experts (MoE) has emerged as a powerful paradigm for scaling model
capacity while preserving computational efficiency. Despite its notable success
in large language models (LLMs), existing attempts to apply MoE to Diffusion
Transformers (DiTs) have yielded limited gains. We attribute this gap to
fundamental differences between language and visual tokens. Language tokens are
semantically dense with pronounced inter-token variation, while visual tokens
exhibit spatial redundancy and functional heterogeneity, hindering expert
specialization in vision MoE. To this end, we present ProMoE, an MoE framework
featuring a two-step router with explicit routing guidance that promotes expert
specialization. Specifically, this guidance encourages the router to partition
image tokens into conditional and unconditional sets via conditional routing
according to their functional roles, and refine the assignments of conditional
image tokens through prototypical routing with learnable prototypes based on
semantic content. Moreover, the similarity-based expert allocation in latent
space enabled by prototypical routing offers a natural mechanism for
incorporating explicit semantic guidance, and we validate that such guidance is
crucial for vision MoE. Building on this, we propose a routing contrastive loss
that explicitly enhances the prototypical routing process, promoting
intra-expert coherence and inter-expert diversity. Extensive experiments on
ImageNet benchmark demonstrate that ProMoE surpasses state-of-the-art methods
under both Rectified Flow and DDPM training objectives. Code and models will be
made publicly available.
ProMoE, an MoE framework with conditional and prototypical routing, enhances expert specialization in Diffusion Transformers, achieving state-of-the-art performance on ImageNet.
Mixture-of-Experts MoE Diffusion Transformers DiTs ProMoE two-step router conditional routing prototypical routing learnable prototypes latent space routing contrastive loss intra-expert coherence inter-expert diversity Rectified Flow DDPM