VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions
Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding.
69c3991c95b48f16a300794b
Adrian Bulat
claimed_verified
2026-03-25T10:22:43.128Z
64a6e0923987f4dd3c37087d
/avatars/88bc86ddce1b38a012b07c81f5c61183.svg
Adrian Bulat
adrianb1
user
adrianb1
69c3991c95b48f16a300794c
Alberto Baldrati
claimed_verified
2026-03-25T10:22:41.031Z
61eeb1121e886efb9a8f57e3
https://cdn-avatars.huggingface.co/v1/production/uploads/1643032839402-noauth.jpeg
Alberto Baldrati
ABaldrati
user
ABaldrati
69c3991c95b48f16a300794d
Ioannis Maniadis Metaxas
69c3991c95b48f16a300794e
Yassine Ouali
69c3991c95b48f16a300794f
Georgios Tzimiropoulos
VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions
Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding.
VISION On Request (VISOR) improves Large Vision-Language Model efficiency by sparsifying image-text interactions through strategic attention layer placement, enabling dynamic visual computation allocation without sacrificing performance.
Large Vision-Language Models visual token reduction information bottleneck visual token sparsification cross-attention self-attention layers dynamic allocation computational budget visual reasoning attention mechanisms