Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.
69a1712ea06f8b911c5d9b5e
Tilemachos Aravanis
69a1712ea06f8b911c5d9b5f
Vladan Stojnić
claimed_verified
2026-02-27T16:38:56.416Z
66a3ae59f33ff23e1c027ccd
https://cdn-avatars.huggingface.co/v1/production/uploads/66a3ae59f33ff23e1c027ccd/tZzpESNPnmhty62xhHszF.jpeg
Vladan Stojnic
stojnvla
user
stojnvla
69a1712ea06f8b911c5d9b60
Bill Psomas
claimed_verified
2026-02-27T16:38:59.565Z
626a9b5205fe1cb65720e00e
https://cdn-avatars.huggingface.co/v1/production/uploads/626a9b5205fe1cb65720e00e/hyWcWn_8jVZsu1Yc5Z0R8.png
Bill Psomas
billpsomas
user
billpsomas
69a1712ea06f8b911c5d9b61
Nikos Komodakis
69a1712ea06f8b911c5d9b62
Giorgos Tolias
Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.
Retrieval-augmented test-time adaptation with learned fusion of textual and visual features bridges the performance gap between zero-shot and supervised open-vocabulary segmentation.
open-vocabulary segmentation vision-language models zero-shot recognition pixel-level prediction textual prompts support set test-time adapter per-image classifier learned fusion modality synergy fine-grained segmentation personalized segmentation