InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
The ability for AI agents to "think with images" requires a sophisticated blend of reasoning and perception. However, current open multimodal agents still largely fall short on the reasoning aspect crucial for real-world tasks like analyzing documents with dense charts/diagrams and navigating maps. To address this gap, we introduce O3-Bench, a new benchmark designed to evaluate multimodal reasoning with interleaved attention to visual details. O3-Bench features challenging problems that require agents to piece together subtle visual information from distinct image areas through multi-step reasoning. The problems are highly challenging even for frontier systems like OpenAI o3, which only obtains 40.8% accuracy on O3-Bench. To make progress, we propose InSight-o3, a multi-agent framework consisting of a visual reasoning agent (vReasoner) and a visual search agent (vSearcher) for which we introduce the task of generalized visual search -- locating relational, fuzzy, or conceptual regions described in free-form language, beyond just simple objects or figures in natural images. We then present a multimodal LLM purpose-trained for this task via reinforcement learning. As a plug-and-play agent, our vSearcher empowers frontier multimodal models (as vReasoners), significantly improving their performance on a wide range of benchmarks. This marks a concrete step towards powerful o3-like open systems. Our code and dataset can be found at https://github.com/m-Just/InSight-o3 .
694a4fe4335742716e9323ce
Kaican Li
claimed_verified
2025-12-25T20:47:40.670Z
65d5b967eeb590ea7435ad07
/avatars/ca0a5e123d5da5aca97cfd8a2d07e60e.svg
Kaican Li
m-Just
user
m-Just
694a4fe4335742716e9323cf
Lewei Yao
694a4fe4335742716e9323d0
Jiannan Wu
694a4fe4335742716e9323d1
Tiezheng Yu
694a4fe4335742716e9323d2
Jierun Chen
694a4fe4335742716e9323d3
Haoli Bai
694a4fe4335742716e9323d4
Lu Hou
694a4fe4335742716e9323d5
Lanqing Hong
694a4fe4335742716e9323d6
Wei Zhang
694a4fe4335742716e9323d7
Nevin L. Zhang
InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
The ability for AI agents to "think with images" requires a sophisticated blend of reasoning and perception. However, current open multimodal agents still largely fall short on the reasoning aspect crucial for real-world tasks like analyzing documents with dense charts/diagrams and navigating maps. To address this gap, we introduce O3-Bench, a new benchmark designed to evaluate multimodal reasoning with interleaved attention to visual details. O3-Bench features challenging problems that require agents to piece together subtle visual information from distinct image areas through multi-step reasoning. The problems are highly challenging even for frontier systems like OpenAI o3, which only obtains 40.8% accuracy on O3-Bench. To make progress, we propose InSight-o3, a multi-agent framework consisting of a visual reasoning agent (vReasoner) and a visual search agent (vSearcher) for which we introduce the task of generalized visual search -- locating relational, fuzzy, or conceptual regions described in free-form language, beyond just simple objects or figures in natural images. We then present a multimodal LLM purpose-trained for this task via reinforcement learning. As a plug-and-play agent, our vSearcher empowers frontier multimodal models (as vReasoners), significantly improving their performance on a wide range of benchmarks. This marks a concrete step towards powerful o3-like open systems. Our code and dataset can be found at https://github.com/m-Just/InSight-o3 .
O3-Bench evaluates multimodal reasoning with interleaved attention to visual details, while InSight-o3 uses a multi-agent framework to improve performance through specialized visual search and reasoning tasks.
multimodal agents reasoning perception O3-Bench interleaved attention visual reasoning visual search vReasoner vSearcher generalized visual search multimodal LLM reinforcement learning