Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding
The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.
6931df252d1e5b0a7d84db96
Jialuo Li
claimed_verified
2025-12-05T08:29:59.578Z
6555a56f7cca2ba493f5331a
https://cdn-avatars.huggingface.co/v1/production/uploads/6555a56f7cca2ba493f5331a/d1_JZic4rFi80xe7m7VWs.jpeg
Jialuo Li
Jialuo21
user
Jialuo21
6931df252d1e5b0a7d84db97
Bin Li
6931df252d1e5b0a7d84db98
Jiahao Li
6931df252d1e5b0a7d84db99
Yan Lu
Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding
The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.
DIG, a query-type adaptive frame selection framework, enhances large multimodal models for long-form video understanding by efficiently handling global and localized queries.
Large Multimodal Models LMMs long-form video understanding context lengths computational cost dense video tokens query-aware frame selection global query localized query uniform sampling specialized pipeline frame selection framework DIG