OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention
While humans perceive the world through diverse modalities that operate synergistically to support a holistic understanding of their surroundings, existing omnivideo models still face substantial challenges on audio-visual understanding tasks. In this paper, we propose OmniVideo-R1, a novel reinforced framework that improves mixed-modality reasoning. OmniVideo-R1 empowers models to "think with omnimodal cues" by two key strategies: (1) query-intensive grounding based on self-supervised learning paradigms; and (2) modality-attentive fusion built upon contrastive learning paradigms. Extensive experiments on multiple benchmarks demonstrate that OmniVideo-R1 consistently outperforms strong baselines, highlighting its effectiveness and robust generalization capabilities.
6989c105beecc443208d2838
Zhangquan Chen
6989c105beecc443208d2839
Jiale Tao
6989c105beecc443208d283a
Ruihuang Li
6989c105beecc443208d283b
Yihao Hu
6989c105beecc443208d283c
Ruitao Chen
6989c105beecc443208d283d
Zhantao Yang
6989c105beecc443208d283e
Xinlei Yu
6989c105beecc443208d283f
Haodong Jing
6989c105beecc443208d2840
Manyuan Zhang
6989c105beecc443208d2841
Shuai Shao
6989c105beecc443208d2842
Biao Wang
6989c105beecc443208d2843
Qinglin Lu
6989c105beecc443208d2844
Ruqi Huang
OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention
While humans perceive the world through diverse modalities that operate synergistically to support a holistic understanding of their surroundings, existing omnivideo models still face substantial challenges on audio-visual understanding tasks. In this paper, we propose OmniVideo-R1, a novel reinforced framework that improves mixed-modality reasoning. OmniVideo-R1 empowers models to "think with omnimodal cues" by two key strategies: (1) query-intensive grounding based on self-supervised learning paradigms; and (2) modality-attentive fusion built upon contrastive learning paradigms. Extensive experiments on multiple benchmarks demonstrate that OmniVideo-R1 consistently outperforms strong baselines, highlighting its effectiveness and robust generalization capabilities.
OmniVideo-R1 enhances audio-visual understanding through reinforced frameworks that integrate self-supervised and contrastive learning for multimodal reasoning.
omnivideo models mixed-modality reasoning self-supervised learning contrastive learning query-intensive grounding modality-attentive fusion