JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion
Audio-Visual Foundation Models, which are pretrained to jointly generate sound and visual content, have recently shown an unprecedented ability to model multi-modal generation and editing, opening new opportunities for downstream tasks. Among these tasks, video dubbing could greatly benefit from such priors, yet most existing solutions still rely on complex, task-specific pipelines that struggle in real-world settings. In this work, we introduce a single-model approach that adapts a foundational audio-video diffusion model for video-to-video dubbing via a lightweight LoRA. The LoRA enables the model to condition on an input audio-video while jointly generating translated audio and synchronized facial motion. To train this LoRA, we leverage the generative model itself to synthesize paired multilingual videos of the same speaker. Specifically, we generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half. By leveraging the rich generative prior of the audio-visual model, our approach preserves speaker identity and lip synchronization while remaining robust to complex motion and real-world dynamics. We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines.
697cb1756676f933227060e8
Anthony Chen
697cb1756676f933227060e9
Naomi Ken Korem
697cb1756676f933227060ea
Tavi Halperin
claimed_verified
2026-03-30T09:11:56.604Z
6369676b85fef3ca96e5e2f1
/avatars/aa33e95ca2d9e70a95623459c2fba9f2.svg
Tavi Halperin
tavihalperin
user
tavihalperin
697cb1756676f933227060eb
Matan Ben Yosef
claimed_verified
2026-03-27T11:38:57.173Z
630389a70907b9a115c1af93
https://cdn-avatars.huggingface.co/v1/production/uploads/630389a70907b9a115c1af93/0iGogiSByXVAvBLHiazoA.jpeg
Matan
matanby
user
matanby
697cb1756676f933227060ec
Urska Jelercic
697cb1756676f933227060ed
Ofir Bibi
claimed_verified
2026-04-16T08:22:06.300Z
631f58935ba8c026340b377c
https://cdn-avatars.huggingface.co/v1/production/uploads/631f58935ba8c026340b377c/4yoHLdNE99VBb7ji_Mzzj.jpeg
Ofir Bibi
ofirbibi
user
ofirbibi
697cb1756676f933227060ee
Or Patashnik
697cb1756676f933227060ef
Daniel Cohen-Or
JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion
Audio-Visual Foundation Models, which are pretrained to jointly generate sound and visual content, have recently shown an unprecedented ability to model multi-modal generation and editing, opening new opportunities for downstream tasks. Among these tasks, video dubbing could greatly benefit from such priors, yet most existing solutions still rely on complex, task-specific pipelines that struggle in real-world settings. In this work, we introduce a single-model approach that adapts a foundational audio-video diffusion model for video-to-video dubbing via a lightweight LoRA. The LoRA enables the model to condition on an input audio-video while jointly generating translated audio and synchronized facial motion. To train this LoRA, we leverage the generative model itself to synthesize paired multilingual videos of the same speaker. Specifically, we generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half. By leveraging the rich generative prior of the audio-visual model, our approach preserves speaker identity and lip synchronization while remaining robust to complex motion and real-world dynamics. We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines.
A lightweight LoRA adaptation of an audio-video diffusion model enables high-quality video dubbing with preserved speaker identity and improved lip synchronization through synthetic multilingual video training.
audio-video diffusion model LoRA video-to-video dubbing generative model multilingual videos lip synchronization speaker identity audio-visual foundation models