TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.
691fd2bccce0eb9b6387af6c
Boshen Xu
claimed_verified
2025-11-21T08:59:53.327Z
67c9728a30792f0621bb2c3c
/avatars/0d70477cbd046823d4abb7b4ba85b0a1.svg
Xu
Boshenxx
user
Boshenxx
691fd2bccce0eb9b6387af6d
Zihan Xiao
691fd2bccce0eb9b6387af6e
Jiaze Li
claimed_verified
2025-11-26T09:28:29.377Z
68fa12db3fea62443368a82e
/avatars/e6059edebe9ee6e3b456d5cae1dab94c.svg
jiaze li
williamljz
user
williamljz
691fd2bccce0eb9b6387af6f
Jianzhong Ju
691fd2bccce0eb9b6387af70
Zhenbo Luo
691fd2bccce0eb9b6387af71
Jian Luan
691fd2bccce0eb9b6387af72
Qin Jin
TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.
TimeViper, a hybrid vision-language model using Mamba-Transformer architecture, efficiently processes long videos by transferring and compressing vision tokens to text tokens, achieving state-of-the-art performance while handling over 10,000 frames.
hybrid vision-language model TimeViper Mamba-Transformer state-space models attention mechanisms vision-to-text information aggregation vision tokens text tokens LLM depth vision token redundancy TransV token information transfer module multimodal understanding hybrid model interpretability