Multi-Vector Index Compression in Any Modality
We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos, but its computation and storage costs grow linearly with document length, making it costly for image-, video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC). AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and video (MSR-VTT, MultiVENT 2.0), we show that attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.
699f21fbebfce7fbcca919cd
Hanxiang Qin
699f21fbebfce7fbcca919ce
Alexander Martin
claimed_verified
2026-03-03T10:43:43.583Z
644ffb154f731658826945c8
/avatars/46fa3a15c2b974692bee5b6ed6468be0.svg
Alex Martin
alexmartin1722
user
alexmartin1722
699f21fbebfce7fbcca919cf
Rohan Jha
699f21fbebfce7fbcca919d0
Chunsheng Zuo
699f21fbebfce7fbcca919d1
Reno Kriz
699f21fbebfce7fbcca919d2
Benjamin Van Durme
Multi-Vector Index Compression in Any Modality
We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos, but its computation and storage costs grow linearly with document length, making it costly for image-, video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC). AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and video (MSR-VTT, MultiVENT 2.0), we show that attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.
Attention-guided clustering method for compressing multi-vector document representations in late interaction retrieval tasks shows superior performance compared to other compression techniques across text, visual-document, and video modalities.
late interaction multi-vector retrieval index compression attention-guided clustering sequence resizing memory tokens hierarchical pooling