Can Visual Input Be Compressed? A Visual Token Compression Benchmark for
  Large Multimodal Models
Large multimodal models (LMMs) often suffer from severe inference
inefficiency due to the large number of visual tokens introduced by image
encoders. While recent token compression methods, such as pruning and merging,
have shown promise in reducing redundancy, their evaluation remains fragmented
and inconsistent. In this work, we present UniPruneBench, a unified and
extensible benchmark for visual token pruning in multimodal LLMs. UniPruneBench
provides standardized protocols across six ability dimensions and ten datasets,
covering ten representative compression algorithms and three families of LMMs
(LLaVA-v1.5, Intern-VL3, and Qwen2.5-VL). Beyond task accuracy, it incorporates
system-level metrics such as runtime and prefilling latency to provide a
holistic view. Our experiments uncover several key findings: (1) random pruning
is a surprisingly strong baseline, (2) no single method consistently
outperforms others across scenarios, (3) pruning sensitivity varies
significantly across tasks, with OCR being most vulnerable, and (4) pruning
ratio is the dominant factor governing performance degradation. We believe
UniPruneBench will serve as a reliable foundation for future research on
efficient multimodal modeling.
690aeddcd70e173c84529080
Tianfan Peng
690aeddcd70e173c84529081
Yuntao Du
690aeddcd70e173c84529082
Pengzhou Ji
690aeddcd70e173c84529083
Shijie Dong
690aeddcd70e173c84529084
Kailin Jiang
claimed_verified
2025-11-05T16:24:24.713Z
65745569839aa08899ea5d27
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg
Kailin Jiang
kailinjiang
user
kailinjiang
690aeddcd70e173c84529085
Mingchuan Ma
690aeddcd70e173c84529086
Yijun Tian
690aeddcd70e173c84529087
Jinhe Bi
690aeddcd70e173c84529088
Qian Li
690aeddcd70e173c84529089
Wei Du
690aeddcd70e173c8452908a
Feng Xiao
690aeddcd70e173c8452908b
Lizhen Cui
Can Visual Input Be Compressed? A Visual Token Compression Benchmark for
  Large Multimodal Models
Large multimodal models (LMMs) often suffer from severe inference
inefficiency due to the large number of visual tokens introduced by image
encoders. While recent token compression methods, such as pruning and merging,
have shown promise in reducing redundancy, their evaluation remains fragmented
and inconsistent. In this work, we present UniPruneBench, a unified and
extensible benchmark for visual token pruning in multimodal LLMs. UniPruneBench
provides standardized protocols across six ability dimensions and ten datasets,
covering ten representative compression algorithms and three families of LMMs
(LLaVA-v1.5, Intern-VL3, and Qwen2.5-VL). Beyond task accuracy, it incorporates
system-level metrics such as runtime and prefilling latency to provide a
holistic view. Our experiments uncover several key findings: (1) random pruning
is a surprisingly strong baseline, (2) no single method consistently
outperforms others across scenarios, (3) pruning sensitivity varies
significantly across tasks, with OCR being most vulnerable, and (4) pruning
ratio is the dominant factor governing performance degradation. We believe
UniPruneBench will serve as a reliable foundation for future research on
efficient multimodal modeling.
UniPruneBench is a unified benchmark for evaluating visual token pruning in multimodal LLMs, providing standardized protocols and system-level metrics to assess performance across various tasks and models.
multimodal models visual tokens image encoders token compression pruning merging UniPruneBench ability dimensions datasets compression algorithms LLaVA-v1.5 Intern-VL3 Qwen2.5-VL runtime prefilling latency OCR pruning sensitivity pruning ratio