Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
We present Uni-MoE 2.0 from the Lychee family. As a fully open-source omnimodal large model (OLM), it substantially advances Lychee's Uni-MoE series in language-centric multimodal understanding, reasoning, and generating. Based on the Qwen2.5-7B dense architecture, we build Uni-MoE-2.0-Omni from scratch through three core contributions: dynamic-capacity Mixture-of-Experts (MoE) design, a progressive training strategy enhanced with an iterative reinforcement strategy, and a carefully curated multimodal data matching technique. It is capable of omnimodal understanding, as well as generating images, text, and speech. Architecturally, our new MoE framework balances computational efficiency and capability for 10 cross-modal inputs using shared, routed, and null experts, while our Omni-Modality 3D RoPE ensures spatio-temporal cross-modality alignment in the self-attention layer. For training, following cross-modal pretraining, we use a progressive supervised fine-tuning strategy that activates modality-specific experts and is enhanced by balanced data composition and an iterative GSPO-DPO method to stabilise RL training and improve reasoning. Data-wise, the base model, trained on approximately 75B tokens of open-source multimodal data, is equipped with special speech and image generation tokens, allowing it to learn these generative tasks by conditioning its outputs on linguistic cues. Extensive evaluation across 85 benchmarks demonstrates that our model achieves SOTA or highly competitive performance against leading OLMs, surpassing Qwen2.5-Omni (trained with 1.2T tokens) on over 50 of 76 benchmarks. Key strengths include video understanding (+7% avg. of 8), omnimodallity understanding (+7% avg. of 4), and audiovisual reasoning (+4%). It also advances long-form speech processing (reducing WER by 4.2%) and leads in low-level image processing and controllable generation across 5 metrics.
691be5fc6bfd5965c0fd37d4
Yunxin Li
claimed_verified
2025-11-19T08:40:54.048Z
62fdb01bc1588e1d4c6c1a7c
/avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
Yunxin Li
YunxinLi
user
YunxinLi
691be5fc6bfd5965c0fd37d5
Xinyu Chen
claimed_verified
2025-11-18T14:08:12.997Z
64d9da538767727dff1e8f19
https://cdn-avatars.huggingface.co/v1/production/uploads/64d9da538767727dff1e8f19/i16e6wETkGwCFmPoJMh8Q.jpeg
Xinyu Chen
Ghaser
user
Ghaser
691be5fc6bfd5965c0fd37d6
Shenyuan Jiang
691be5fc6bfd5965c0fd37d7
Haoyuan Shi
claimed_verified
2025-11-18T14:08:19.326Z
652fb8bcc9dd2692a25ef2e3
/avatars/461e6cc1c3441cde18192b080b0b8576.svg
Haoyuan Shi
MrSunshy
user
MrSunshy
691be5fc6bfd5965c0fd37d8
Zhenyu Liu
691be5fc6bfd5965c0fd37d9
Xuanyu Zhang
691be5fc6bfd5965c0fd37da
Nanhao Deng
691be5fc6bfd5965c0fd37db
Zhenran Xu
claimed_verified
2025-11-20T08:50:11.049Z
639c379cdb7c5f35004066cb
/avatars/3e435506ee85aa7d2d0ec2174a07462f.svg
Zhenran Xu
imryanxu
user
imryanxu
691be5fc6bfd5965c0fd37dc
Yicheng Ma
691be5fc6bfd5965c0fd37dd
Meishan Zhang
691be5fc6bfd5965c0fd37de
Baotian Hu
691be5fc6bfd5965c0fd37df
Min Zhang
Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
We present Uni-MoE 2.0 from the Lychee family. As a fully open-source omnimodal large model (OLM), it substantially advances Lychee's Uni-MoE series in language-centric multimodal understanding, reasoning, and generating. Based on the Qwen2.5-7B dense architecture, we build Uni-MoE-2.0-Omni from scratch through three core contributions: dynamic-capacity Mixture-of-Experts (MoE) design, a progressive training strategy enhanced with an iterative reinforcement strategy, and a carefully curated multimodal data matching technique. It is capable of omnimodal understanding, as well as generating images, text, and speech. Architecturally, our new MoE framework balances computational efficiency and capability for 10 cross-modal inputs using shared, routed, and null experts, while our Omni-Modality 3D RoPE ensures spatio-temporal cross-modality alignment in the self-attention layer. For training, following cross-modal pretraining, we use a progressive supervised fine-tuning strategy that activates modality-specific experts and is enhanced by balanced data composition and an iterative GSPO-DPO method to stabilise RL training and improve reasoning. Data-wise, the base model, trained on approximately 75B tokens of open-source multimodal data, is equipped with special speech and image generation tokens, allowing it to learn these generative tasks by conditioning its outputs on linguistic cues. Extensive evaluation across 85 benchmarks demonstrates that our model achieves SOTA or highly competitive performance against leading OLMs, surpassing Qwen2.5-Omni (trained with 1.2T tokens) on over 50 of 76 benchmarks. Key strengths include video understanding (+7% avg. of 8), omnimodallity understanding (+7% avg. of 4), and audiovisual reasoning (+4%). It also advances long-form speech processing (reducing WER by 4.2%) and leads in low-level image processing and controllable generation across 5 metrics.