Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
Segmentation based on language has been a popular topic in computer vision. While recent advances in multimodal large language models (MLLMs) have endowed segmentation systems with reasoning capabilities, these efforts remain confined by the frozen internal knowledge of MLLMs, which limits their potential for real-world scenarios that involve up-to-date information or domain-specific concepts. In this work, we propose Seg-ReSearch, a novel segmentation paradigm that overcomes the knowledge bottleneck of existing approaches. By enabling interleaved reasoning and external search, Seg-ReSearch empowers segmentation systems to handle dynamic, open-world queries that extend beyond the frozen knowledge of MLLMs. To effectively train this capability, we introduce a hierarchical reward design that harmonizes initial guidance with progressive incentives, mitigating the dilemma between sparse outcome signals and rigid step-wise supervision. For evaluation, we construct OK-VOS, a challenging benchmark that explicitly requires outside knowledge for video object segmentation. Experiments on OK-VOS and two existing reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin. Code and data will be released at https://github.com/iSEE-Laboratory/Seg-ReSearch.
6985bb7b4ad556f294b7ed95
Tianming Liang
claimed_verified
2026-02-06T18:50:11.951Z
66c98c27fafc0fc87c280749
/avatars/c71db3bee0fcd9aabcc38fd871d1cb75.svg
Tianming Liang
liangtm
user
liangtm
6985bb7b4ad556f294b7ed96
Qirui Du
6985bb7b4ad556f294b7ed97
Jian-Fang Hu
6985bb7b4ad556f294b7ed98
Haichao Jiang
6985bb7b4ad556f294b7ed99
Zicheng Lin
6985bb7b4ad556f294b7ed9a
Wei-Shi Zheng
Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
Segmentation based on language has been a popular topic in computer vision. While recent advances in multimodal large language models (MLLMs) have endowed segmentation systems with reasoning capabilities, these efforts remain confined by the frozen internal knowledge of MLLMs, which limits their potential for real-world scenarios that involve up-to-date information or domain-specific concepts. In this work, we propose Seg-ReSearch, a novel segmentation paradigm that overcomes the knowledge bottleneck of existing approaches. By enabling interleaved reasoning and external search, Seg-ReSearch empowers segmentation systems to handle dynamic, open-world queries that extend beyond the frozen knowledge of MLLMs. To effectively train this capability, we introduce a hierarchical reward design that harmonizes initial guidance with progressive incentives, mitigating the dilemma between sparse outcome signals and rigid step-wise supervision. For evaluation, we construct OK-VOS, a challenging benchmark that explicitly requires outside knowledge for video object segmentation. Experiments on OK-VOS and two existing reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin. Code and data will be released at https://github.com/iSEE-Laboratory/Seg-ReSearch.
Seg-ReSearch introduces a novel segmentation approach that combines interleaved reasoning with external search to overcome limitations of frozen MLLM knowledge, using hierarchical reward design for training and demonstrating superior performance on video object segmentation benchmarks.
multimodal large language models segmentation interleaved reasoning external search knowledge bottleneck hierarchical reward design video object segmentation OK-VOS benchmark