JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
AIGC has rapidly expanded from text-to-image generation toward high-quality multimodal synthesis across video and audio. Within this context, joint audio-video generation (JAVG) has emerged as a fundamental task that produces synchronized and semantically aligned sound and vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and alignment with human preferences. To bridge the gap, this paper presents JavisDiT++, a concise yet powerful framework for unified modeling and optimization of JAVG. First, we introduce a modality-specific mixture-of-experts (MS-MoE) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose a temporal-aligned RoPE (TA-RoPE) strategy to achieve explicit, frame-level synchronization between audio and video tokens. Besides, we develop an audio-video direct preference optimization (AV-DPO) method to align model outputs with human preference across quality, consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2V, our model achieves state-of-the-art performance merely with around 1M public training entries, significantly outperforming prior approaches in both qualitative and quantitative evaluations. Comprehensive ablation studies have been conducted to validate the effectiveness of our proposed modules. All the code, model, and dataset are released at https://JavisVerse.github.io/JavisDiT2-page.
699fb334ebfce7fbcca91ad0
Kai Liu
699fb334ebfce7fbcca91ad1
Yanhao Zheng
699fb334ebfce7fbcca91ad2
Kai Wang
699fb334ebfce7fbcca91ad3
Shengqiong Wu
699fb334ebfce7fbcca91ad4
Rongjunchen Zhang
699fb334ebfce7fbcca91ad5
Jiebo Luo
699fb334ebfce7fbcca91ad6
Dimitrios Hatzinakos
699fb334ebfce7fbcca91ad7
Ziwei Liu
699fb334ebfce7fbcca91ad8
Hao Fei
claimed_verified
2026-03-09T10:39:25.488Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
scofield7419
user
scofield7419
699fb334ebfce7fbcca91ad9
Tat-Seng Chua
JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
AIGC has rapidly expanded from text-to-image generation toward high-quality multimodal synthesis across video and audio. Within this context, joint audio-video generation (JAVG) has emerged as a fundamental task that produces synchronized and semantically aligned sound and vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and alignment with human preferences. To bridge the gap, this paper presents JavisDiT++, a concise yet powerful framework for unified modeling and optimization of JAVG. First, we introduce a modality-specific mixture-of-experts (MS-MoE) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose a temporal-aligned RoPE (TA-RoPE) strategy to achieve explicit, frame-level synchronization between audio and video tokens. Besides, we develop an audio-video direct preference optimization (AV-DPO) method to align model outputs with human preference across quality, consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2V, our model achieves state-of-the-art performance merely with around 1M public training entries, significantly outperforming prior approaches in both qualitative and quantitative evaluations. Comprehensive ablation studies have been conducted to validate the effectiveness of our proposed modules. All the code, model, and dataset are released at https://JavisVerse.github.io/JavisDiT2-page.
JavisDiT++ presents a unified framework for joint audio-video generation using modality-specific mixture-of-experts, temporal-aligned RoPE, and audio-video direct preference optimization to achieve high-quality, synchronized multimedia synthesis.
joint audio-video generation modality-specific mixture-of-experts MS-MoE temporal-aligned RoPE TA-RoPE audio-video direct preference optimization AV-DPO multimodal synthesis synchronized generation semantically aligned