AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding
Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token
69cb7f2c22f9e9dce3fe2268
Haozhe Qi
claimed_verified
2026-03-31T09:23:50.137Z
65998a6ce1604bf996604217
/avatars/adc94c8097018f73cf18ed628aa7d507.svg
Haozhe Qi
haozheqi
user
haozheqi
69cb7f2c22f9e9dce3fe2269
Kevin Qu
claimed_verified
2026-04-01T08:36:05.773Z
660ae64e20634eb93dfafe2b
/avatars/9099f45f05e65574e435fe16189da060.svg
Kevin Qu
KevinQu7
user
KevinQu7
69cb7f2c22f9e9dce3fe226a
Mahdi Rad
69cb7f2c22f9e9dce3fe226b
Rui Wang
69cb7f2c22f9e9dce3fe226c
Alexander Mathis
69cb7f2c22f9e9dce3fe226d
Marc Pollefeys
AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding
Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token
AdaptToken enables efficient long-video understanding by using model uncertainty to dynamically select relevant tokens across video segments, achieving improved accuracy and reduced inference time through global budget allocation and early stopping mechanisms.
Multi-modal Large Language Models video understanding token selection self-uncertainty cross-modal attention entropy early stopping global token budget allocation