Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks, but they face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams. In streaming video scenarios, the primary bottleneck lies in the Vision Transformer (ViT) encoding stage, where redundant processing of temporally similar frames leads to inefficiency. Additionally, inflated token sequences during LLM pre-filling further exacerbate latency and memory overhead. To address these challenges, we propose Streaming Token Compression (STC), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs, optimizing both ViT encoding and LLM pre-filling stages to accelerate processing. STC introduces two token-level accelerators: STC-Cacher, which reduces ViT encoding overhead by caching and reusing features from temporally similar frames, and STC-Pruner, which compresses the visual token sequence before it enters the LLM, preserving only the most salient tokens based on both spatial and temporal relevance. Extensive experiments on four baseline streaming VideoLLMs across five benchmarks demonstrate that STC outperforms other compression methods. Notably, STC retains up to 99\% of accuracy on the ReKV framework while reducing ViT encoding latency and LLM pre-filling latency by 24.5\% and 45.3\%.
692e876937312eaa83fd892b
Yiyu Wang
692e876937312eaa83fd892c
Xuyang Liu
claimed_verified
2025-12-18T08:21:59.084Z
66a0caa1a7a6ed88ad1c0ddf
https://cdn-avatars.huggingface.co/v1/production/uploads/66a0caa1a7a6ed88ad1c0ddf/WoOP24-ruuHy4ryNhRp0D.jpeg
Xuyang Liu
xuyang-liu16
user
xuyang-liu16
692e876937312eaa83fd892d
Xiyan Gui
692e876937312eaa83fd892e
Xinying Lin
692e876937312eaa83fd892f
Boxue Yang
claimed_verified
2025-12-02T16:50:20.079Z
688c72c011ef3399b561dee7
https://cdn-avatars.huggingface.co/v1/production/uploads/688c72c011ef3399b561dee7/puhgnTOAfZYetsC46hqGm.jpeg
BoxueYang
Boxue
user
Boxue
692e876937312eaa83fd8930
Chenfei Liao
claimed_verified
2025-12-02T16:50:18.006Z
6806464ed918f6d2fee2bc8b
https://cdn-avatars.huggingface.co/v1/production/uploads/6806464ed918f6d2fee2bc8b/rgpG2oO0m6PT0KltCF_Wf.jpeg
Chenfei Liao
Chenfei-Liao
user
Chenfei-Liao
692e876937312eaa83fd8931
Tailai Chen
692e876937312eaa83fd8932
Linfeng Zhang
Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks, but they face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams. In streaming video scenarios, the primary bottleneck lies in the Vision Transformer (ViT) encoding stage, where redundant processing of temporally similar frames leads to inefficiency. Additionally, inflated token sequences during LLM pre-filling further exacerbate latency and memory overhead. To address these challenges, we propose Streaming Token Compression (STC), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs, optimizing both ViT encoding and LLM pre-filling stages to accelerate processing. STC introduces two token-level accelerators: STC-Cacher, which reduces ViT encoding overhead by caching and reusing features from temporally similar frames, and STC-Pruner, which compresses the visual token sequence before it enters the LLM, preserving only the most salient tokens based on both spatial and temporal relevance. Extensive experiments on four baseline streaming VideoLLMs across five benchmarks demonstrate that STC outperforms other compression methods. Notably, STC retains up to 99\% of accuracy on the ReKV framework while reducing ViT encoding latency and LLM pre-filling latency by 24.5\% and 45.3\%.
STC, a hierarchical framework, optimizes streaming VideoLLMs by reducing ViT encoding and LLM pre-filling latency through token caching and pruning, without significantly impacting accuracy.
Vision Transformer ViT Streaming Video Large Language Models VideoLLMs token compression STC-Cacher STC-Pruner ReKV framework