OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding
Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often lack the fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, a fully audio-guided active perception agent that dynamically orchestrates specialized tools to achieve more fine-grained audio-visual reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, this paper demonstrates a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and proprietary models by substantial margins of 10% - 20% accuracy.
695349a689916ff627aa3f47
Keda Tao
695349a689916ff627aa3f48
Wenjie Du
claimed_verified
2026-03-23T11:02:25.677Z
6624ce8248e016b5ea4ba952
https://cdn-avatars.huggingface.co/v1/production/uploads/6624ce8248e016b5ea4ba952/WbpwxXABS5gomQJtSiiCo.jpeg
wenjiedu
Kurt232
user
Kurt232
695349a689916ff627aa3f49
Bohan Yu
695349a689916ff627aa3f4a
Weiqiang Wang
695349a689916ff627aa3f4b
Jian Liu
695349a689916ff627aa3f4c
Huan Wang
OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding
Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often lack the fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, a fully audio-guided active perception agent that dynamically orchestrates specialized tools to achieve more fine-grained audio-visual reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, this paper demonstrates a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and proprietary models by substantial margins of 10% - 20% accuracy.