Unified Spatio-Temporal Token Scoring for Efficient Video VLMs
Token pruning is essential for enhancing the computational efficiency of vision-language models (VLMs), particularly for video-based tasks where temporal redundancy is prevalent. Prior approaches typically prune tokens either (1) within the vision transformer (ViT) exclusively for unimodal perception tasks such as action recognition and object segmentation, without adapting to downstream vision-language tasks; or (2) only within the LLM while leaving the ViT output intact, often requiring complex text-conditioned token selection mechanisms. In this paper, we introduce Spatio-Temporal Token Scoring (STTS), a simple and lightweight module that prunes vision tokens across both the ViT and the LLM without text conditioning or token merging, and is fully compatible with end-to-end training. By learning how to score temporally via an auxiliary loss and spatially via LLM downstream gradients, aided by our efficient packing algorithm, STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks. Efficiency gains increase with more sampled frames per video. Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline. Overall, STTS represents a novel, simple yet effective technique for unified, architecture-wide vision token pruning.
69bb59b2ba2b62a5fb0b0bce
Jianrui Zhang
claimed_verified
2026-03-23T10:56:52.009Z
6508b164abdde5290e5e4939
https://cdn-avatars.huggingface.co/v1/production/uploads/6508b164abdde5290e5e4939/lQgAs3BHwCyI7Go1QA62m.jpeg
Harris Zhang
HanSolo9682
user
HanSolo9682
69bb59b2ba2b62a5fb0b0bcf
Yue Yang
claimed_verified
2026-04-07T08:47:50.892Z
62f6c68904e5e02f82b04690
https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
Yue Yang
yyupenn
user
yyupenn
69bb59b2ba2b62a5fb0b0bd0
Rohun Tripathi
69bb59b2ba2b62a5fb0b0bd1
Winson Han
69bb59b2ba2b62a5fb0b0bd2
Ranjay Krishna
69bb59b2ba2b62a5fb0b0bd3
Christopher Clark
69bb59b2ba2b62a5fb0b0bd4
Yong Jae Lee
69bb59b2ba2b62a5fb0b0bd5
Sangho Lee
Unified Spatio-Temporal Token Scoring for Efficient Video VLMs
Token pruning is essential for enhancing the computational efficiency of vision-language models (VLMs), particularly for video-based tasks where temporal redundancy is prevalent. Prior approaches typically prune tokens either (1) within the vision transformer (ViT) exclusively for unimodal perception tasks such as action recognition and object segmentation, without adapting to downstream vision-language tasks; or (2) only within the LLM while leaving the ViT output intact, often requiring complex text-conditioned token selection mechanisms. In this paper, we introduce Spatio-Temporal Token Scoring (STTS), a simple and lightweight module that prunes vision tokens across both the ViT and the LLM without text conditioning or token merging, and is fully compatible with end-to-end training. By learning how to score temporally via an auxiliary loss and spatially via LLM downstream gradients, aided by our efficient packing algorithm, STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks. Efficiency gains increase with more sampled frames per video. Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline. Overall, STTS represents a novel, simple yet effective technique for unified, architecture-wide vision token pruning.
STTS is a lightweight module that efficiently prunes vision tokens across both vision transformer and language model architectures in vision-language models for video tasks, achieving significant computational gains with minimal performance loss.
token pruning vision-language models vision transformer language model spatio-temporal token scoring auxiliary loss downstream gradients end-to-end training video-based tasks temporal redundancy vision tokens