ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning
Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy-cost learning signal. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to 16x more frames at the same visual budget while delivering over 15% performance gain. Code is available at https://github.com/Xnhyacinth/ResAdapt.
69cb372c22f9e9dce3fe1f8f
Huanxuan Liao
admin_assigned
2026-03-31T09:41:31.913Z
634900f08146350b3a4b6710
https://cdn-avatars.huggingface.co/v1/production/uploads/1665728713794-noauth.jpeg
Huanxuan Liao
Xnhyacinth
user
Xnhyacinth
69cb372c22f9e9dce3fe1f90
Zhongtao Jiang
69cb372c22f9e9dce3fe1f91
Yupu Hao
69cb372c22f9e9dce3fe1f92
Yuqiao Tan
69cb372c22f9e9dce3fe1f93
Shizhu He
69cb372c22f9e9dce3fe1f94
Jun Zhao
69cb372c22f9e9dce3fe1f95
Kun Xu
69cb372c22f9e9dce3fe1f96
Kang Liu
ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning
Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy-cost learning signal. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to 16x more frames at the same visual budget while delivering over 15% performance gain. Code is available at https://github.com/Xnhyacinth/ResAdapt.
ResAdapt is an input-side adaptation framework that dynamically allocates visual resources to improve multimodal large language models' efficiency in video tasks while maintaining high performance.
Multimodal Large Language Models visual token growth spatial resolution temporal context ResAdapt Input-side adaptation framework Allocator Cost-Aware Policy Optimization contextual bandit visual budget video QA temporal grounding image reasoning