OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
69841218e34659da7e1f4df3
Yue Ding
claimed_verified
2026-02-05T10:54:01.167Z
66100bacac50abb8d56dece6
https://cdn-avatars.huggingface.co/v1/production/uploads/66100bacac50abb8d56dece6/fd-4VMpb_1nl903yAIK4K.jpeg
Ding Yue
dingyue1011
user
dingyue1011
69841218e34659da7e1f4df4
Yiyan Ji
69841218e34659da7e1f4df5
Jungang Li
claimed_verified
2026-02-05T10:54:09.654Z
64b76528fdb702b3d8641514
https://cdn-avatars.huggingface.co/v1/production/uploads/64b76528fdb702b3d8641514/Ho-uWcQCAEIURM1lhWEWJ.jpeg
Jungang Li
Jungang
user
Jungang
69841218e34659da7e1f4df6
Xuyang Liu
claimed_verified
2026-02-05T10:54:06.046Z
66a0caa1a7a6ed88ad1c0ddf
https://cdn-avatars.huggingface.co/v1/production/uploads/66a0caa1a7a6ed88ad1c0ddf/WoOP24-ruuHy4ryNhRp0D.jpeg
Xuyang Liu
xuyang-liu16
user
xuyang-liu16
69841218e34659da7e1f4df7
Xinlong Chen
69841218e34659da7e1f4df8
Junfei Wu
69841218e34659da7e1f4df9
Bozhou Li
claimed_verified
2026-04-08T08:07:16.601Z
661e62c6bac5d981f886f77b
/avatars/f1eb51ed4499ca434c8939573dfbd5e2.svg
Bozhou Li
zooblastlbz
user
zooblastlbz
69841218e34659da7e1f4dfa
Bohan Zeng
claimed_verified
2026-04-07T08:49:17.092Z
6671214c92412fd4640714eb
/avatars/48fa84e7bc3bb92ad0192aa26b32de10.svg
bohan zeng
zbhpku
user
zbhpku
69841218e34659da7e1f4dfb
Yang Shi
claimed_verified
2026-02-18T09:06:40.798Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
DogNeverSleep
user
DogNeverSleep
69841218e34659da7e1f4dfc
Yushuo Guan
claimed_verified
2026-02-05T10:54:04.081Z
66ac46766c3f950f4f10b9f9
/avatars/027b573bc6e5b18107e762645cec6069.svg
Yushuo Guan
UnnamedWatcher
user
UnnamedWatcher
69841218e34659da7e1f4dfd
Yuanxing Zhang
69841218e34659da7e1f4dfe
Jiaheng Liu
69841218e34659da7e1f4dff
Qiang Liu
69841218e34659da7e1f4e00
Pengfei Wan
69841218e34659da7e1f4e01
Liang Wang
OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
OmniSIFT is a modality-asymmetric token compression framework for Omni-LLMs that reduces computational overhead through spatio-temporal video pruning and vision-guided audio selection while maintaining superior performance.
Omni-modal Large Language Models token compression spatio-temporal video pruning vision-guided audio selection differentiable straight-through estimator end-to-end optimization multimodal token sequences