YingVideo-MV: Music-Driven Multi-Stage Video Generation
While diffusion model for audio-driven avatar video generation have achieved notable process in synthesizing long sequences with natural audio-visual synchronization and identity consistency, the generation of music-performance videos with camera motions remains largely unexplored. We present YingVideo-MV, the first cascaded framework for music-driven long-video generation. Our approach integrates audio semantic analysis, an interpretable shot planning module (MV-Director), temporal-aware diffusion Transformer architectures, and long-sequence consistency modeling to enable automatic synthesis of high-quality music performance videos from audio signals. We construct a large-scale Music-in-the-Wild Dataset by collecting web data to support the achievement of diverse, high-quality results. Observing that existing long-video generation methods lack explicit camera motion control, we introduce a camera adapter module that embeds camera poses into latent noise. To enhance continulity between clips during long-sequence inference, we further propose a time-aware dynamic window range strategy that adaptively adjust denoising ranges based on audio embedding. Comprehensive benchmark tests demonstrate that YingVideo-MV achieves outstanding performance in generating coherent and expressive music videos, and enables precise music-motion-camera synchronization. More videos are available in our project page: https://giantailab.github.io/YingVideo-MV/ .
692faf0326742347f61dac52
Jiahui Chen
claimed_verified
2025-12-05T08:30:09.750Z
6888798cc609a7453fa1f8a4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tMnLhl9Q12_ZjjnfpKVye.png
陈佳辉
CaffeyChen
user
CaffeyChen
692faf0326742347f61dac53
Weida Wang
692faf0326742347f61dac54
Runhua Shi
692faf0326742347f61dac55
Huan Yang
692faf0326742347f61dac56
Chaofan Ding
692faf0326742347f61dac57
Zihao Chen
YingVideo-MV: Music-Driven Multi-Stage Video Generation
While diffusion model for audio-driven avatar video generation have achieved notable process in synthesizing long sequences with natural audio-visual synchronization and identity consistency, the generation of music-performance videos with camera motions remains largely unexplored. We present YingVideo-MV, the first cascaded framework for music-driven long-video generation. Our approach integrates audio semantic analysis, an interpretable shot planning module (MV-Director), temporal-aware diffusion Transformer architectures, and long-sequence consistency modeling to enable automatic synthesis of high-quality music performance videos from audio signals. We construct a large-scale Music-in-the-Wild Dataset by collecting web data to support the achievement of diverse, high-quality results. Observing that existing long-video generation methods lack explicit camera motion control, we introduce a camera adapter module that embeds camera poses into latent noise. To enhance continulity between clips during long-sequence inference, we further propose a time-aware dynamic window range strategy that adaptively adjust denoising ranges based on audio embedding. Comprehensive benchmark tests demonstrate that YingVideo-MV achieves outstanding performance in generating coherent and expressive music videos, and enables precise music-motion-camera synchronization. More videos are available in our project page: https://giantailab.github.io/YingVideo-MV/ .
YingVideo-MV generates high-quality music performance videos with synchronized camera motion using cascaded frameworks, audio semantic analysis, and temporal-aware diffusion Transformers.
diffusion model audio-driven avatar video generation audio semantic analysis shot planning module MV-Director temporal-aware diffusion Transformer architectures long-sequence consistency modeling Music-in-the-Wild Dataset camera adapter module latent noise time-aware dynamic window range strategy denoising ranges audio embedding music-motion-camera synchronization