UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal
  Interactions
Due to the lack of effective cross-modal modeling, existing open-source
audio-video generation methods often exhibit compromised lip synchronization
and insufficient semantic consistency. To mitigate these drawbacks, we propose
UniAVGen, a unified framework for joint audio and video generation. UniAVGen is
anchored in a dual-branch joint synthesis architecture, incorporating two
parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent
space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which
enables bidirectional, temporally aligned cross-attention, thus ensuring
precise spatiotemporal synchronization and semantic consistency. Furthermore,
this cross-modal interaction is augmented by a Face-Aware Modulation module,
which dynamically prioritizes salient regions in the interaction process. To
enhance generative fidelity during inference, we additionally introduce
Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly
amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint
synthesis design enables seamless unification of pivotal audio-video tasks
within a single model, such as joint audio-video generation and continuation,
video-to-audio dubbing, and audio-driven video synthesis. Comprehensive
experiments validate that, with far fewer training samples (1.3M vs. 30.1M),
UniAVGen delivers overall advantages in audio-video synchronization, timbre
consistency, and emotion consistency.
690c3bd460494e4fa76756d6
Guozhen Zhang
claimed_verified
2025-11-06T13:45:45.325Z
6684152a443492c24cdac044
/avatars/1d5abbde12a808aa743769603e494ddb.svg
Guozhen Zhang
zgzaacm
user
zgzaacm
690c3bd460494e4fa76756d7
Zixiang Zhou
690c3bd460494e4fa76756d8
Teng Hu
claimed_verified
2025-11-27T10:15:25.970Z
6484576450e57f3530ee6432
/avatars/16a780f94597e9f5a4d57a84879102d7.svg
huteng
JTUplayer
user
JTUplayer
690c3bd460494e4fa76756d9
Ziqiao Peng
690c3bd460494e4fa76756da
Youliang Zhang
claimed_verified
2026-02-03T10:10:10.252Z
67d3a681536175c33a85c7ed
/avatars/7fd5969c7eed2127f45bf5a9f4189280.svg
zliang
youliang1233214
user
youliang1233214
690c3bd460494e4fa76756db
Yi Chen
690c3bd460494e4fa76756dc
Yuan Zhou
690c3bd460494e4fa76756dd
Qinglin Lu
690c3bd460494e4fa76756de
Limin Wang
claimed_verified
2025-11-06T13:45:43.289Z
62c77f4352d8ae531f5511f9
/avatars/50198ccb02ccd286975a4613fbabee28.svg
Limin Wang
lmwang
user
lmwang
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal
  Interactions
Due to the lack of effective cross-modal modeling, existing open-source
audio-video generation methods often exhibit compromised lip synchronization
and insufficient semantic consistency. To mitigate these drawbacks, we propose
UniAVGen, a unified framework for joint audio and video generation. UniAVGen is
anchored in a dual-branch joint synthesis architecture, incorporating two
parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent
space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which
enables bidirectional, temporally aligned cross-attention, thus ensuring
precise spatiotemporal synchronization and semantic consistency. Furthermore,
this cross-modal interaction is augmented by a Face-Aware Modulation module,
which dynamically prioritizes salient regions in the interaction process. To
enhance generative fidelity during inference, we additionally introduce
Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly
amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint
synthesis design enables seamless unification of pivotal audio-video tasks
within a single model, such as joint audio-video generation and continuation,
video-to-audio dubbing, and audio-driven video synthesis. Comprehensive
experiments validate that, with far fewer training samples (1.3M vs. 30.1M),
UniAVGen delivers overall advantages in audio-video synchronization, timbre
consistency, and emotion consistency.
UniAVGen, a unified framework using dual Diffusion Transformers and Asymmetric Cross-Modal Interaction, enhances audio-video generation by ensuring synchronization and consistency with fewer training samples.
Diffusion Transformers DiTs cross-modal latent space Asymmetric Cross-Modal Interaction cross-attention Face-Aware Modulation Modality-Aware Classifier-Free Guidance audio-video generation video-to-audio dubbing audio-driven video synthesis