DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation
Recent advancements in foundation models have revolutionized joint audio-video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-video generation (R2AV), video editing (RV2AV) and audio-driven video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-video generation. Specifically, we design a Symmetric Conditional Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models. We will release our code to bridge the gap between academic research and commercial-grade applications.
698e9f3bcace060ff123ae16
Xu Guo
claimed_verified
2026-02-26T10:03:53.038Z
67d50738fed7787297d737d6
/avatars/30d7126f7c3feda732c5783ea3db9c7f.svg
xuguo
XuGuo699
user
XuGuo699
698e9f3bcace060ff123ae17
Fulong Ye
claimed_verified
2026-04-16T08:21:31.298Z
6339029a76421c0543167075
https://cdn-avatars.huggingface.co/v1/production/uploads/6339029a76421c0543167075/3npT0NxTMV-MLWDThyBy8.png
fulong ye
Alon77777
user
Alon77777
698e9f3bcace060ff123ae18
Qichao Sun
claimed_verified
2026-02-27T16:44:49.123Z
674566cb79d6f3a9da7be0de
/avatars/b6a5384820e150405039aa2b9badac29.svg
Qichao Sun
Simons212
user
Simons212
698e9f3bcace060ff123ae19
Liyang Chen
claimed_verified
2026-04-16T08:21:33.190Z
65e9866cdde63935299cead7
https://cdn-avatars.huggingface.co/v1/production/uploads/65e9866cdde63935299cead7/yBF_iDl0ga6lfMp3w-TTz.jpeg
Liyang Chen
leoniuschen
user
leoniuschen
698e9f3bcace060ff123ae1a
Bingchuan Li
claimed_verified
2026-03-06T13:57:55.287Z
63b415037af2e415f2599c18
https://cdn-avatars.huggingface.co/v1/production/uploads/63b415037af2e415f2599c18/KjlY5jZxdPznmf9qW5k3W.jpeg
Bingchuan Li
lbc402
user
lbc402
698e9f3bcace060ff123ae1b
Pengze Zhang
698e9f3bcace060ff123ae1c
Jiawei Liu
698e9f3bcace060ff123ae1d
Songtao Zhao
698e9f3bcace060ff123ae1e
Qian He
698e9f3bcace060ff123ae1f
Xiangwang Hou
DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation
Recent advancements in foundation models have revolutionized joint audio-video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-video generation (R2AV), video editing (RV2AV) and audio-driven video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-video generation. Specifically, we design a Symmetric Conditional Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models. We will release our code to bridge the gap between academic research and commercial-grade applications.
DreamID-Omni is a unified framework for controllable human-centric audio-video generation that uses a symmetric conditional diffusion transformer with dual-level disentanglement and multi-task progressive training to achieve state-of-the-art performance.
conditional diffusion transformer symmetric conditional injection scheme dual-level disentanglement synchronized RoPE structured captions multi-task progressive training audio-video generation reference-based audio-video generation video editing audio-driven video animation identity-timbre binding speaker confusion diffusion models