Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: https://fr0zencrane.github.io/uni-vigu-page/.
69ddde74aea31e227c22c643
Luozheng Qin
claimed_verified
2026-04-14T08:14:56.404Z
66a9b3533d417b0baa9220a6
/avatars/48e2915bcd2ee254fa0f9cdba54cab92.svg
Luozheng Qin
Fr0zencr4nE
user
Fr0zencr4nE
69ddde74aea31e227c22c644
Jia Gong
69ddde74aea31e227c22c645
Qian Qiao
claimed_verified
2026-04-14T08:14:54.310Z
656975b88931d28b2444163d
https://cdn-avatars.huggingface.co/v1/production/uploads/656975b88931d28b2444163d/EdMB7SCQGtd8PGquouQ4H.jpeg
joeqian
joeqianqian
user
joeqianqian
69ddde74aea31e227c22c646
Tianjiao Li
69ddde74aea31e227c22c647
Li Xu
69ddde74aea31e227c22c648
Haoyu Pan
69ddde74aea31e227c22c649
Chao Qu
69ddde74aea31e227c22c64a
Zhiyu Tan
69ddde74aea31e227c22c64b
Hao Li
Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: https://fr0zencrane.github.io/uni-vigu-page/.
Uni-ViGU presents a generation-centric approach to unified multimodal video understanding and generation by extending video generation as a foundation through unified flow matching and bidirectional training mechanisms.
multimodal models video generation video understanding unified flow method continuous flow matching discrete flow matching modality-driven MoE Transformer blocks bidirectional training Knowledge Recall Capability Refinement