OmniGAIA: Towards Native Omni-Modal AI Agents
Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.
69a0fda9a13deaa4494488e3
Xiaoxi Li
69a0fda9a13deaa4494488e4
Wenxiang Jiao
claimed_verified
2026-02-27T16:41:37.304Z
63db16330cc3bc12bc0b6f8f
https://cdn-avatars.huggingface.co/v1/production/uploads/63db16330cc3bc12bc0b6f8f/ld0JQIfX1SBlDVDOmw9VT.jpeg
Wenxiang Jiao
wxjiao
user
wxjiao
69a0fda9a13deaa4494488e5
Jiarui Jin
claimed_verified
2026-03-31T09:31:40.842Z
646cc5add2cc138217dbe133
https://cdn-avatars.huggingface.co/v1/production/uploads/646cc5add2cc138217dbe133/Uzvt9_UWIRIT4t4gHdjid.jpeg
Jiarui Jin
jinjiarui97
user
jinjiarui97
69a0fda9a13deaa4494488e6
Shijian Wang
claimed_verified
2026-03-31T09:31:38.882Z
67f5454e1f54f65efc9ce06b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QTh1c-okqsl-NLHatc8Gu.png
Shijian Wang
ShijianW01
user
ShijianW01
69a0fda9a13deaa4494488e7
Guanting Dong
claimed_verified
2026-02-27T16:41:09.826Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
dongguanting
user
dongguanting
69a0fda9a13deaa4494488e8
Jiajie Jin
69a0fda9a13deaa4494488e9
Hao Wang
69a0fda9a13deaa4494488ea
Yinuo Wang
69a0fda9a13deaa4494488eb
Ji-Rong Wen
69a0fda9a13deaa4494488ec
Yuan Lu
69a0fda9a13deaa4494488ed
Zhicheng Dou
OmniGAIA: Towards Native Omni-Modal AI Agents
Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.
OmniGAIA benchmark evaluates multi-modal agents on complex reasoning tasks across video, audio, and image modalities, while OmniAtlas agent improves tool-use capabilities through hindsight-guided tree exploration and OmniDPO fine-tuning.
multi-modal LLMs omni-modal perception cross-modal reasoning tool-integrated reasoning hindsight-guided tree exploration OmniDPO