Small Vision-Language Models are Smart Compressors for Long Video Understanding
Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free O(1) dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.
69d87dca79548974da94660a
Junjie Fei
claimed_verified
2026-04-10T13:34:02.100Z
66d7ded3d9b727201f85c54e
/avatars/99c579ce4f7faae2924793180a674013.svg
Junjie Fei
FeiElysia
user
FeiElysia
69d87dca79548974da94660b
Jun Chen
69d87dca79548974da94660c
Zechun Liu
69d87dca79548974da94660d
Yunyang Xiong
69d87dca79548974da94660e
Chong Zhou
69d87dca79548974da94660f
Wei Wen
69d87dca79548974da946610
Junlin Han
69d87dca79548974da946611
Mingchen Zhuge
69d87dca79548974da946612
Saksham Suri
69d87dca79548974da946613
Qi Qian
69d87dca79548974da946614
Shuming Liu
claimed_verified
2026-04-13T07:33:35.793Z
65803d64defc9c0d30db4f88
https://cdn-avatars.huggingface.co/v1/production/uploads/65803d64defc9c0d30db4f88/gVxJaB8vSUFj4r0vnTVbP.jpeg
Shuming Liu
sming256
user
sming256
69d87dca79548974da946615
Lemeng Wu
69d87dca79548974da946616
Raghuraman Krishnamoorthi
69d87dca79548974da946617
Vikas Chandra
69d87dca79548974da946618
Mohamed Elhoseiny
69d87dca79548974da946619
Chenchen Zhu
Small Vision-Language Models are Smart Compressors for Long Video Understanding
Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free O(1) dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.
Tempo is an efficient framework that compresses long videos for multimodal understanding by using a small vision-language model for temporal compression and adaptive token allocation to maintain intent-aligned representations within strict budgets.
Multimodal Large Language Models visual streams token budgets lost-in-the-middle phenomenon dense visual streams sparse sampling uniform pooling Small Vision-Language Model cross-modal distillation Adaptive Token Allocation temporal compression intent-aligned representations causal consistency dynamic routing long-form video understanding