Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers
In the retrieval domain, candidates' fusion from heterogeneous retrievers is
a long-standing challenge, particularly for complex, multi-modal data such as
videos. While typical fusion techniques are training-free, they rely solely on
rank or score signals, disregarding candidates' representations. This work
introduces Vote-in-Context (ViC), a generalized, training-free framework that
re-thinks list-wise reranking and fusion as a zero-shot reasoning task for a
Vision-Language Model (VLM). The core insight is to serialize both content
evidence and retriever metadata directly within the VLM's prompt, allowing the
model to adaptively weigh retriever consensus against visual-linguistic
content. We demonstrate the generality of this framework by applying it to the
challenging domain of cross-modal video retrieval. To this end, we introduce
the S-Grid, a compact serialization map that represents each video as an image
grid, optionally paired with subtitles to enable list-wise reasoning over video
candidates. ViC is evaluated both as a single-list reranker, where it
dramatically improves the precision of individual retrievers, and as an
ensemble fuser, where it consistently outperforms strong baselines like
CombSUM. Across video retrieval benchmarks including ActivityNet and VATEX, the
framework establishes new state-of-the-art zero-shot retrieval performance,
demonstrating its effectiveness in handling complex visual and temporal signals
alongside text. In zero-shot settings, ViC achieves Recall@1 scores of 87.1%
(t2v) / 89.0% (v2t) on MSR-VTT and 99.6% (v2t) on VATEX, representing massive
gains of up to +40 Recall@1 over previous state-of-the-art baselines. We
present ViC as a simple, reproducible, and highly effective recipe for turning
modern VLMs into powerful zero-shot rerankers and fusers. Code and resources
are publicly available at: https://github.com/mohammad2012191/ViC
6909c945812eca10f9cc6471
Mohamed Eltahir
claimed_verified
2025-11-04T10:49:03.133Z
6331c242e092098b57bd8e58
/avatars/8bcaf3cb3482a002ded96d3206b04947.svg
Mohamed Eltahir
mohammad2012191
user
mohammad2012191
6909c945812eca10f9cc6472
Ali Habibullah
6909c945812eca10f9cc6473
Lama Ayash
6909c945812eca10f9cc6474
Tanveer Hussain
6909c945812eca10f9cc6475
Naeemullah Khan
Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers
In the retrieval domain, candidates' fusion from heterogeneous retrievers is
a long-standing challenge, particularly for complex, multi-modal data such as
videos. While typical fusion techniques are training-free, they rely solely on
rank or score signals, disregarding candidates' representations. This work
introduces Vote-in-Context (ViC), a generalized, training-free framework that
re-thinks list-wise reranking and fusion as a zero-shot reasoning task for a
Vision-Language Model (VLM). The core insight is to serialize both content
evidence and retriever metadata directly within the VLM's prompt, allowing the
model to adaptively weigh retriever consensus against visual-linguistic
content. We demonstrate the generality of this framework by applying it to the
challenging domain of cross-modal video retrieval. To this end, we introduce
the S-Grid, a compact serialization map that represents each video as an image
grid, optionally paired with subtitles to enable list-wise reasoning over video
candidates. ViC is evaluated both as a single-list reranker, where it
dramatically improves the precision of individual retrievers, and as an
ensemble fuser, where it consistently outperforms strong baselines like
CombSUM. Across video retrieval benchmarks including ActivityNet and VATEX, the
framework establishes new state-of-the-art zero-shot retrieval performance,
demonstrating its effectiveness in handling complex visual and temporal signals
alongside text. In zero-shot settings, ViC achieves Recall@1 scores of 87.1%
(t2v) / 89.0% (v2t) on MSR-VTT and 99.6% (v2t) on VATEX, representing massive
gains of up to +40 Recall@1 over previous state-of-the-art baselines. We
present ViC as a simple, reproducible, and highly effective recipe for turning
modern VLMs into powerful zero-shot rerankers and fusers. Code and resources
are publicly available at: https://github.com/mohammad2012191/ViC
Vote-in-Context (ViC) is a training-free framework that leverages Vision-Language Models (VLMs) for zero-shot reranking and fusion in cross-modal video retrieval, achieving state-of-the-art performance.
Vote-in-Context ViC Vision-Language Model VLM list-wise reranking fusion zero-shot reasoning S-Grid image grid subtitles list-wise reasoning ensemble fuser CombSUM ActivityNet VATEX Recall@1 zero-shot retrieval