Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
We introduce MoS (Mixture of States), a novel fusion paradigm for multimodal diffusion models that merges modalities using flexible, state-based interactions. The core of MoS is a learnable, token-wise router that creates denoising timestep- and input-dependent interactions between modalities' hidden states, precisely aligning token-level features with the diffusion trajectory. This router sparsely selects the top-k hidden states and is trained with an ε-greedy strategy, efficiently selecting contextual features with minimal learnable parameters and negligible computational overhead. We validate our design with text-to-image generation (MoS-Image) and editing (MoS-Editing), which achieve state-of-the-art results. With only 3B to 5B parameters, our models match or surpass counterparts up to 4times larger. These findings establish MoS as a flexible and compute-efficient paradigm for scaling multimodal diffusion models.
691ef72c3c64d32b03645a18
Haozhe Liu
691ef72c3c64d32b03645a19
Ding Liu
691ef72c3c64d32b03645a1a
Mingchen Zhuge
691ef72c3c64d32b03645a1b
Zijian Zhou
claimed_verified
2025-12-09T10:34:24.874Z
62e0fcfe4db2175cd2710063
/avatars/0f8d1b5ba3e3eba2af18f98f65f0b180.svg
Zijian Zhou
franciszzj
user
franciszzj
691ef72c3c64d32b03645a1c
Tian Xie
691ef72c3c64d32b03645a1d
Sen He
691ef72c3c64d32b03645a1e
Yukang Yang
691ef72c3c64d32b03645a1f
Shuming Liu
claimed_verified
2026-01-29T09:18:05.069Z
65803d64defc9c0d30db4f88
https://cdn-avatars.huggingface.co/v1/production/uploads/65803d64defc9c0d30db4f88/gVxJaB8vSUFj4r0vnTVbP.jpeg
Shuming Liu
sming256
user
sming256
691ef72c3c64d32b03645a20
Yuren Cong
691ef72c3c64d32b03645a21
Jiadong Guo
691ef72c3c64d32b03645a22
Hongyu Xu
691ef72c3c64d32b03645a23
Ke Xu
691ef72c3c64d32b03645a24
Kam-Woh Ng
691ef72c3c64d32b03645a25
Juan C. Pérez
691ef72c3c64d32b03645a26
Juan-Manuel~Pérez-Rúa
691ef72c3c64d32b03645a27
Tao Xiang
691ef72c3c64d32b03645a28
Wei Liu
691ef72c3c64d32b03645a29
Shikun Liu
691ef72c3c64d32b03645a2a
Jürgen Schmidhuber
Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
We introduce MoS (Mixture of States), a novel fusion paradigm for multimodal diffusion models that merges modalities using flexible, state-based interactions. The core of MoS is a learnable, token-wise router that creates denoising timestep- and input-dependent interactions between modalities' hidden states, precisely aligning token-level features with the diffusion trajectory. This router sparsely selects the top-k hidden states and is trained with an ε-greedy strategy, efficiently selecting contextual features with minimal learnable parameters and negligible computational overhead. We validate our design with text-to-image generation (MoS-Image) and editing (MoS-Editing), which achieve state-of-the-art results. With only 3B to 5B parameters, our models match or surpass counterparts up to 4times larger. These findings establish MoS as a flexible and compute-efficient paradigm for scaling multimodal diffusion models.
MoS, a novel multimodal diffusion model fusion paradigm, achieves state-of-the-art results in text-to-image generation and editing with minimal parameters and computational overhead by using a learnable, token-wise router for modality interaction.
MoS Mixture of States multimodal diffusion models token-wise router denoising timestep input-dependent interactions hidden states diffusion trajectory top-k hidden states ε-greedy strategy contextual features compute-efficient paradigm