Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy
The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization. To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.
69283a32805584b280405089
Teng Hu
69283a32805584b28040508a
Zhentao Yu
69283a32805584b28040508b
Guozhen Zhang
claimed_verified
2025-11-28T13:48:52.007Z
6684152a443492c24cdac044
/avatars/1d5abbde12a808aa743769603e494ddb.svg
Guozhen Zhang
zgzaacm
user
zgzaacm
69283a32805584b28040508c
Zihan Su
69283a32805584b28040508d
Zhengguang Zhou
69283a32805584b28040508e
Youliang Zhang
claimed_verified
2026-02-03T10:09:49.386Z
67d3a681536175c33a85c7ed
/avatars/7fd5969c7eed2127f45bf5a9f4189280.svg
zliang
youliang1233214
user
youliang1233214
69283a32805584b28040508f
Yuan Zhou
69283a32805584b280405090
Qinglin Lu
69283a32805584b280405091
Ran Yi
Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy
The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization. To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.
Harmony addresses audio-visual synchronization in generative AI by introducing a Cross-Task Synergy training paradigm, Global-Local Decoupled Interaction Module, and Synchronization-Enhanced CFG to improve alignment and fidelity.
joint diffusion process Correspondence Drift global attention mechanisms temporal cues intra-modal bias Classifier-Free Guidance Cross-Task Synergy Global-Local Decoupled Interaction Module Synchronization-Enhanced CFG