InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
Window attention and linear attention represent two principal strategies for mitigating the quadratic complexity and ever-growing KV cache in Vision-Language Models (VLMs). However, we observe that window-based VLMs suffer performance degradation when sequence length exceeds the window size, while linear attention underperforms on information-intensive tasks such as OCR and document understanding. To overcome these limitations, we propose InfiniteVL, a linear-complexity VLM architecture that synergizes sliding window attention (SWA) with Gated DeltaNet. For achieving competitive multimodal performance under constrained resources, we design a three-stage training strategy comprising distillation pretraining, instruction tuning, and long-sequence SFT. Remarkably, using less than 2\% of the training data required by leading VLMs, InfiniteVL not only substantially outperforms previous linear-complexity VLMs but also matches the performance of leading Transformer-based VLMs, while demonstrating effective long-term memory retention. Compared to similar-sized Transformer-based VLMs accelerated by FlashAttention-2, InfiniteVL achieves over 3.6\times inference speedup while maintaining constant latency and memory footprint. In streaming video understanding scenarios, it sustains a stable 24 FPS real-time prefill speed while preserving long-term memory cache. Code and models are available at https://github.com/hustvl/InfiniteVL.
693a1f0274fced5bf9c32399
Hongyuan Tao
admin_assigned
2025-12-11T10:46:24.642Z
66a105bb456284adf458d656
/avatars/b543a324f7e159d6e84bc68915e93d24.svg
Tao Hongyuan
HongyuanTao
user
HongyuanTao
693a1f0274fced5bf9c3239a
Bencheng Liao
admin_assigned
2025-12-11T10:46:30.879Z
6577073fc2bf55b1f6bafb49
/avatars/58803398b1a918b7570db17893e65122.svg
Bencheng liao
LegendBC
user
LegendBC
693a1f0274fced5bf9c3239b
Shaoyu Chen
693a1f0274fced5bf9c3239c
Haoran Yin
693a1f0274fced5bf9c3239d
Qian Zhang
693a1f0274fced5bf9c3239e
Wenyu Liu
admin_assigned
2025-12-11T10:46:45.682Z
66c2e7fc934e2f07753542ac
/avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
LWY
wenyuliu
user
wenyuliu
693a1f0274fced5bf9c3239f
Xinggang Wang
admin_assigned
2025-12-11T10:46:51.670Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
xinggangw
user
xinggangw
InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
Window attention and linear attention represent two principal strategies for mitigating the quadratic complexity and ever-growing KV cache in Vision-Language Models (VLMs). However, we observe that window-based VLMs suffer performance degradation when sequence length exceeds the window size, while linear attention underperforms on information-intensive tasks such as OCR and document understanding. To overcome these limitations, we propose InfiniteVL, a linear-complexity VLM architecture that synergizes sliding window attention (SWA) with Gated DeltaNet. For achieving competitive multimodal performance under constrained resources, we design a three-stage training strategy comprising distillation pretraining, instruction tuning, and long-sequence SFT. Remarkably, using less than 2\% of the training data required by leading VLMs, InfiniteVL not only substantially outperforms previous linear-complexity VLMs but also matches the performance of leading Transformer-based VLMs, while demonstrating effective long-term memory retention. Compared to similar-sized Transformer-based VLMs accelerated by FlashAttention-2, InfiniteVL achieves over 3.6\times inference speedup while maintaining constant latency and memory footprint. In streaming video understanding scenarios, it sustains a stable 24 FPS real-time prefill speed while preserving long-term memory cache. Code and models are available at https://github.com/hustvl/InfiniteVL.
InfiniteVL, a linear-complexity VLM architecture combining sliding window attention and Gated DeltaNet, achieves competitive performance with less data and faster inference than leading Transformer-based models.
window attention linear attention Vision-Language Models VLMs sequence length window size sliding window attention Gated DeltaNet distillation pretraining instruction tuning long-sequence SFT multimodal performance long-term memory retention FlashAttention-2 inference speedup real-time prefill speed