SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video
  Diffusion Transformers
The generation and editing of audio-conditioned talking portraits guided by
multimodal inputs, including text, images, and videos, remains under explored.
In this paper, we present SkyReels-Audio, a unified framework for synthesizing
high-fidelity and temporally coherent talking portrait videos. Built upon
pretrained video diffusion transformers, our framework supports infinite-length
generation and editing, while enabling diverse and controllable conditioning
through multimodal inputs. We employ a hybrid curriculum learning strategy to
progressively align audio with facial motion, enabling fine-grained multimodal
control over long video sequences. To enhance local facial coherence, we
introduce a facial mask loss and an audio-guided classifier-free guidance
mechanism. A sliding-window denoising approach further fuses latent
representations across temporal segments, ensuring visual fidelity and temporal
consistency across extended durations and diverse identities. More importantly,
we construct a dedicated data pipeline for curating high-quality triplets
consisting of synchronized audio, video, and textual descriptions.
Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior
performance in lip-sync accuracy, identity consistency, and realistic facial
dynamics, particularly under complex and challenging conditions.
684264d1a9584289f0053f5c
Zhengcong Fei
684264d1a9584289f0053f5d
Hao Jiang
684264d1a9584289f0053f5e
Di Qiu
claimed_verified
2025-06-06T07:40:29.181Z
65bef422fdb8d33cefeaccc3
/avatars/d40b0d7dda21fa1a68c291d11bc357ec.svg
Qiu Di
diqiu7
user
diqiu7
684264d1a9584289f0053f5f
Baoxuan Gu
684264d1a9584289f0053f60
Youqiang Zhang
684264d1a9584289f0053f61
Jiahua Wang
684264d1a9584289f0053f62
Jialin Bai
684264d1a9584289f0053f63
Debang Li
684264d1a9584289f0053f64
Mingyuan Fan
684264d1a9584289f0053f65
Guibin Chen
684264d1a9584289f0053f66
Yahui Zhou
SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video
  Diffusion Transformers
The generation and editing of audio-conditioned talking portraits guided by
multimodal inputs, including text, images, and videos, remains under explored.
In this paper, we present SkyReels-Audio, a unified framework for synthesizing
high-fidelity and temporally coherent talking portrait videos. Built upon
pretrained video diffusion transformers, our framework supports infinite-length
generation and editing, while enabling diverse and controllable conditioning
through multimodal inputs. We employ a hybrid curriculum learning strategy to
progressively align audio with facial motion, enabling fine-grained multimodal
control over long video sequences. To enhance local facial coherence, we
introduce a facial mask loss and an audio-guided classifier-free guidance
mechanism. A sliding-window denoising approach further fuses latent
representations across temporal segments, ensuring visual fidelity and temporal
consistency across extended durations and diverse identities. More importantly,
we construct a dedicated data pipeline for curating high-quality triplets
consisting of synchronized audio, video, and textual descriptions.
Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior
performance in lip-sync accuracy, identity consistency, and realistic facial
dynamics, particularly under complex and challenging conditions.
SkyReels-Audio is a unified framework using pretrained video diffusion transformers for generating high-fidelity and coherent audio-conditioned talking portrait videos, supported by a hybrid curriculum learning strategy and advanced loss mechanisms.
video diffusion transformers infinite-length generation multimodal inputs hybrid curriculum learning facial mask loss classifier-free guidance mechanism sliding-window denoising lip-sync accuracy identity consistency realistic facial dynamics