Bridging Your Imagination with Audio-Video Generation via a Unified Director
Existing AI-driven video creation systems typically treat script drafting and key-shot design as two disjoint tasks: the former relies on large language models, while the latter depends on image generation models. We argue that these two tasks should be unified within a single framework, as logical reasoning and imaginative thinking are both fundamental qualities of a film director. In this work, we propose UniMAGE, a unified director model that bridges user prompts with well-structured scripts, thereby empowering non-experts to produce long-context, multi-shot films by leveraging existing audio-video generation models. To achieve this, we employ the Mixture-of-Transformers architecture that unifies text and image generation. To further enhance narrative logic and keyframe consistency, we introduce a ``first interleaving, then disentangling'' training paradigm. Specifically, we first perform Interleaved Concept Learning, which utilizes interleaved text-image data to foster the model's deeper understanding and imaginative interpretation of scripts. We then conduct Disentangled Expert Learning, which decouples script writing from keyframe generation, enabling greater flexibility and creativity in storytelling. Extensive experiments demonstrate that UniMAGE achieves state-of-the-art performance among open-source models, generating logically coherent video scripts and visually consistent keyframe images.
69535b9989916ff627aa4031
Jiaxu Zhang
69535b9989916ff627aa4032
Tianshu Hu
69535b9989916ff627aa4033
Yuan Zhang
claimed_verified
2026-01-28T11:33:01.078Z
66b0719b30ad67d64328cf95
/avatars/bec39dcbb4eccdbf84b303996d74d87a.svg
Yuan
necrazy
user
necrazy
69535b9989916ff627aa4034
Zenan Li
69535b9989916ff627aa4035
Linjie Luo
69535b9989916ff627aa4036
Guosheng Lin
69535b9989916ff627aa4037
Xin Chen
Bridging Your Imagination with Audio-Video Generation via a Unified Director
Existing AI-driven video creation systems typically treat script drafting and key-shot design as two disjoint tasks: the former relies on large language models, while the latter depends on image generation models. We argue that these two tasks should be unified within a single framework, as logical reasoning and imaginative thinking are both fundamental qualities of a film director. In this work, we propose UniMAGE, a unified director model that bridges user prompts with well-structured scripts, thereby empowering non-experts to produce long-context, multi-shot films by leveraging existing audio-video generation models. To achieve this, we employ the Mixture-of-Transformers architecture that unifies text and image generation. To further enhance narrative logic and keyframe consistency, we introduce a ``first interleaving, then disentangling'' training paradigm. Specifically, we first perform Interleaved Concept Learning, which utilizes interleaved text-image data to foster the model's deeper understanding and imaginative interpretation of scripts. We then conduct Disentangled Expert Learning, which decouples script writing from keyframe generation, enabling greater flexibility and creativity in storytelling. Extensive experiments demonstrate that UniMAGE achieves state-of-the-art performance among open-source models, generating logically coherent video scripts and visually consistent keyframe images.
UniMAGE is a unified director model that combines script drafting and key-shot design using a Mixture-of-Transformers architecture with interleaved concept learning and disentangled expert learning to create coherent video scripts and consistent keyframe images.
Mixture-of-Transformers interleaved text-image data interleaved concept learning disentangled expert learning script writing keyframe generation narrative logic visual consistency