Col-Bandit: Zero-Shot Query-Time Pruning for Late-Interaction Retrieval
Multi-vector late-interaction retrievers such as ColBERT achieve state-of-the-art retrieval quality, but their query-time cost is dominated by exhaustively computing token-level MaxSim interactions for every candidate document. While approximating late interaction with single-vector representations reduces cost, it often incurs substantial accuracy loss. We introduce Col-Bandit, a query-time pruning algorithm that reduces this computational burden by casting reranking as a finite-population Top-K identification problem. Col-Bandit maintains uncertainty-aware bounds over partially observed document scores and adaptively reveals only the (document, query token) MaxSim entries needed to determine the top results under statistical decision bounds with a tunable relaxation. Unlike coarse-grained approaches that prune entire documents or tokens offline, Col-Bandit sparsifies the interaction matrix on the fly. It operates as a zero-shot, drop-in layer over standard multi-vector systems, requiring no index modifications, offline preprocessing, or model retraining. Experiments on textual (BEIR) and multimodal (REAL-MM-RAG) benchmarks show that Col-Bandit preserves ranking fidelity while reducing MaxSim FLOPs by up to 5times, indicating that dense late-interaction scoring contains substantial redundancy that can be identified and pruned efficiently at query time.
698af29c1b2dc6b37d61b322
Roi Pony
claimed_verified
2026-02-10T09:04:03.510Z
67b63b7828019ee3d38ee8fb
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/oqVtzM8kQ6ZokMP7Qj0us.png
roi pony
roi-pony
user
roi-pony
698af29c1b2dc6b37d61b323
Adi Raz
claimed_verified
2026-02-10T09:04:00.895Z
67ab764de48a46f097ffebf2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/68zlE-pEQAJHv9CdaPSJz.png
Raz Goldfarb
Adirazgold
user
Adirazgold
698af29c1b2dc6b37d61b324
Oshri Naparstek
698af29c1b2dc6b37d61b325
Idan Friedman
claimed_verified
2026-03-19T13:05:01.352Z
66efd1134ed3540120d128d5
/avatars/038ff620903ee3dbcee565433ccc8aa1.svg
idan friedman
idanfriedman
user
idanfriedman
698af29c1b2dc6b37d61b326
Udi Barzelay
claimed_verified
2026-03-18T08:46:52.387Z
66323035d9fb78439d3a3b80
/avatars/04754ec424869b5f543260a8b2f36acf.svg
Barzelay
Udibarzi
user
Udibarzi
Col-Bandit: Zero-Shot Query-Time Pruning for Late-Interaction Retrieval
Multi-vector late-interaction retrievers such as ColBERT achieve state-of-the-art retrieval quality, but their query-time cost is dominated by exhaustively computing token-level MaxSim interactions for every candidate document. While approximating late interaction with single-vector representations reduces cost, it often incurs substantial accuracy loss. We introduce Col-Bandit, a query-time pruning algorithm that reduces this computational burden by casting reranking as a finite-population Top-K identification problem. Col-Bandit maintains uncertainty-aware bounds over partially observed document scores and adaptively reveals only the (document, query token) MaxSim entries needed to determine the top results under statistical decision bounds with a tunable relaxation. Unlike coarse-grained approaches that prune entire documents or tokens offline, Col-Bandit sparsifies the interaction matrix on the fly. It operates as a zero-shot, drop-in layer over standard multi-vector systems, requiring no index modifications, offline preprocessing, or model retraining. Experiments on textual (BEIR) and multimodal (REAL-MM-RAG) benchmarks show that Col-Bandit preserves ranking fidelity while reducing MaxSim FLOPs by up to 5times, indicating that dense late-interaction scoring contains substantial redundancy that can be identified and pruned efficiently at query time.
Col-Bandit reduces computational costs in multi-vector late-interaction retrieval by adaptively pruning token-level interactions during query processing while maintaining ranking accuracy.
multi-vector late-interaction retrievers ColBERT MaxSim interactions reranking Top-K identification finite-population uncertainty-aware bounds query-time pruning sparse interaction matrix dense late-interaction scoring