JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for Joint Audio-Video (JAV) comprehension and generation. JavisGPT adopts a concise encoder-LLM-decoder architecture, featuring a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. To support this, we further construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that span diverse and multi-level comprehension and generation scenarios. Extensive experiments on JAV comprehension and generation benchmarks show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.
6953792489916ff627aa4099
Kai Liu
6953792489916ff627aa409a
Jungang Li
claimed_verified
2026-01-02T15:40:30.222Z
64b76528fdb702b3d8641514
https://cdn-avatars.huggingface.co/v1/production/uploads/64b76528fdb702b3d8641514/Ho-uWcQCAEIURM1lhWEWJ.jpeg
Jungang Li
Jungang
user
Jungang
6953792489916ff627aa409b
Yuchong Sun
6953792489916ff627aa409c
Shengqiong Wu
6953792489916ff627aa409d
Jianzhang Gao
6953792489916ff627aa409e
Daoan Zhang
6953792489916ff627aa409f
Wei Zhang
6953792489916ff627aa40a0
Sheng Jin
6953792489916ff627aa40a1
Sicheng Yu
6953792489916ff627aa40a2
Geng Zhan
6953792489916ff627aa40a3
Jiayi Ji
claimed_verified
2026-01-12T10:36:54.374Z
67f0043781f4f7a1b4386f80
/avatars/5e4667bcb3a89552a0fd74a5a398b2a1.svg
Jiayi Ji
Yolov999
user
Yolov999
6953792489916ff627aa40a4
Fan Zhou
6953792489916ff627aa40a5
Liang Zheng
6953792489916ff627aa40a6
Shuicheng Yan
6953792489916ff627aa40a7
Hao Fei
claimed_verified
2026-01-02T15:40:31.921Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
scofield7419
user
scofield7419
6953792489916ff627aa40a8
Tat-Seng Chua
JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for Joint Audio-Video (JAV) comprehension and generation. JavisGPT adopts a concise encoder-LLM-decoder architecture, featuring a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. To support this, we further construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that span diverse and multi-level comprehension and generation scenarios. Extensive experiments on JAV comprehension and generation benchmarks show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.