UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation
Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: https://github.com/dvlab-research/UnityVideo
6937cb9719d912300c34a470
Jiehui Huang
6937cb9719d912300c34a471
Yuechen Zhang
claimed_verified
2026-03-04T09:25:23.234Z
6418554a0956be7233a1023e
https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
zhang yuechen
julianjuaner
user
julianjuaner
6937cb9719d912300c34a472
Xu He
claimed_verified
2026-01-02T15:40:50.047Z
66a356f2f7352f4ffbb4e74a
/avatars/4391990768ffc079cf9146973614ba84.svg
He
Phoebux
user
Phoebux
6937cb9719d912300c34a473
Yuan Gao
6937cb9719d912300c34a474
Zhi Cen
6937cb9719d912300c34a475
Bin Xia
6937cb9719d912300c34a476
Yan Zhou
6937cb9719d912300c34a477
Xin Tao
6937cb9719d912300c34a478
Pengfei Wan
6937cb9719d912300c34a479
Jiaya Jia
UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation
Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: https://github.com/dvlab-research/UnityVideo
UnityVideo, a unified framework, enhances video generation by integrating multiple modalities and paradigms, leading to improved quality and alignment with real-world constraints.
UnityVideo dynamic noising modality switcher in-context learner segmentation masks human skeletons DensePose optical flow depth maps joint optimization zero-shot generalization