Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.
69b378b72ad6c5d4cf486a59
Baifeng Shi
claimed_verified
2026-03-26T14:04:32.373Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
bfshi
user
bfshi
69b378b72ad6c5d4cf486a5a
Stephanie Fu
69b378b72ad6c5d4cf486a5b
Long Lian
69b378b72ad6c5d4cf486a5c
Hanrong Ye
69b378b72ad6c5d4cf486a5d
David Eigen
69b378b72ad6c5d4cf486a5e
Aaron Reite
69b378b72ad6c5d4cf486a5f
Boyi Li
69b378b72ad6c5d4cf486a60
Jan Kautz
69b378b72ad6c5d4cf486a61
Song Han
69b378b72ad6c5d4cf486a62
David M. Chan
69b378b72ad6c5d4cf486a63
Pavlo Molchanov
69b378b72ad6c5d4cf486a64
Trevor Darrell
69b378b72ad6c5d4cf486a65
Hongxu Yin
Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.
AutoGaze is a lightweight module that reduces redundant video patches before processing by vision transformers or multi-modal large language models, enabling efficient processing of long, high-resolution videos while maintaining performance.
multi-modal large language models vision transformers next-token prediction reinforcement learning autoregressive selection visual tokens video understanding long-form video QA high-resolution videos