Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering research, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.
69ae2f409893f58272f740c6
Lijiang Li
69ae2f409893f58272f740c7
Zuwei Long
69ae2f409893f58272f740c8
Yunhang Shen
claimed_verified
2026-04-08T08:07:00.797Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
shenyunhang
user
shenyunhang
69ae2f409893f58272f740c9
Heting Gao
69ae2f409893f58272f740ca
Haoyu Cao
claimed_verified
2026-04-08T11:40:44.319Z
641c1a85999935676ec7ead4
/avatars/738a4e43bb3ddd4de287acfa26095b3d.svg
caohaoyu
rechy
user
rechy
69ae2f409893f58272f740cb
Xing Sun
claimed_verified
2026-04-03T09:09:28.424Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
69ae2f409893f58272f740cc
Caifeng Shan
69ae2f409893f58272f740cd
Ran He
69ae2f409893f58272f740ce
Chaoyou Fu
admin_assigned
2026-03-11T08:54:08.341Z
653b268cd1041ca9188954da
https://cdn-avatars.huggingface.co/v1/production/uploads/653b268cd1041ca9188954da/5Qy0fyQdRCIUUAzV6fg1k.png
Chaoyou Fu
BradyFU
user
BradyFU
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering research, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.
Omni-Diffusion introduces the first any-to-any multimodal language model based on mask-based discrete diffusion models, unifying text, speech, and image processing in a single framework.
multimodal large language models autoregressive architecture discrete diffusion models mask-based discrete diffusion models multimodal systems discrete multimodal tokens bimodal tasks multimodal foundation models