Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
Recent audio-video generative systems suggest that coupling modalities benefits not only audio-video synchrony but also the video modality itself. We pose a fundamental question: Does audio-video joint denoising training improve video generation, even when we only care about video quality? To study this, we introduce a parameter-efficient Audio-Video Full DiT (AVFullDiT) architecture that leverages pre-trained text-to-video (T2V) and text-to-audio (T2A) modules for joint denoising. We train (i) a T2AV model with AVFullDiT and (ii) a T2V-only counterpart under identical settings. Our results provide the first systematic evidence that audio-video joint denoising can deliver more than synchrony. We observe consistent improvements on challenging subsets featuring large and object contact motions. We hypothesize that predicting audio acts as a privileged signal, encouraging the model to internalize causal relationships between visual events and their acoustic consequences (e.g., collision times impact sound), which in turn regularizes video dynamics. Our findings suggest that cross-modal co-training is a promising approach to developing stronger, more physically grounded world models. Code and dataset will be made publicly available.
692f9ccf26742347f61daad8
Jianzong Wu
692f9ccf26742347f61daad9
Hao Lian
692f9ccf26742347f61daada
Dachao Hao
692f9ccf26742347f61daadb
Ye Tian
692f9ccf26742347f61daadc
Qingyu Shi
claimed_verified
2025-12-03T09:15:58.766Z
656724074f6ec72017754d33
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zLIgt4xvfbYPLZh0E3WWF.png
QingyuShi
QingyuShi
user
QingyuShi
692f9ccf26742347f61daadd
Biaolong Chen
692f9ccf26742347f61daade
Hao Jiang
Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
Recent audio-video generative systems suggest that coupling modalities benefits not only audio-video synchrony but also the video modality itself. We pose a fundamental question: Does audio-video joint denoising training improve video generation, even when we only care about video quality? To study this, we introduce a parameter-efficient Audio-Video Full DiT (AVFullDiT) architecture that leverages pre-trained text-to-video (T2V) and text-to-audio (T2A) modules for joint denoising. We train (i) a T2AV model with AVFullDiT and (ii) a T2V-only counterpart under identical settings. Our results provide the first systematic evidence that audio-video joint denoising can deliver more than synchrony. We observe consistent improvements on challenging subsets featuring large and object contact motions. We hypothesize that predicting audio acts as a privileged signal, encouraging the model to internalize causal relationships between visual events and their acoustic consequences (e.g., collision times impact sound), which in turn regularizes video dynamics. Our findings suggest that cross-modal co-training is a promising approach to developing stronger, more physically grounded world models. Code and dataset will be made publicly available.
Audio-video joint denoising improves video generation quality by regularizing video dynamics through audio as a privileged signal, as demonstrated by the AVFullDiT architecture.
Audio-Video Full DiT AVFullDiT text-to-video text-to-audio T2AV T2V joint denoising cross-modal co-training world models