HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering
Long-form video question answering requires reasoning over extended temporal contexts, making frame selection critical for large vision-language models (LVLMs) bound by finite context windows. Existing methods face a sharp trade-off: similarity-based selectors are fast but collapse compositional queries into a single dense vector, losing sub-event ordering and cross-modal bindings; agent-based methods recover this structure through iterative LVLM inference, but at prohibitive cost. We introduce HiMu, a training-free framework that bridges this gap. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (ASR, CLAP). The resulting signals are normalized, temporally smoothed to align different modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, producing a continuous satisfaction curve. Evaluations on Video-MME, LongVideoBench and HERBench-Lite show that HiMu advances the efficiency-accuracy Pareto front: at 16 frames with Qwen3-VL 8B it outperforms all competing selectors, and with GPT-4o it surpasses agentic systems operating at 32-512 frames while requiring roughly 10x fewer FLOPs.
69c0e8033fd3fef6237a68b7
Dan Ben-Ami
69c0e8033fd3fef6237a68b8
Gabriele Serussi
claimed_verified
2026-03-23T09:15:55.926Z
66214a94bbe70ad73f95b99b
/avatars/fb16480dbb4eab9e1324cc73259a276c.svg
Gabriele Serussi
GSerussi
user
GSerussi
69c0e8033fd3fef6237a68b9
Kobi Cohen
69c0e8033fd3fef6237a68ba
Chaim Baskin
HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering
Long-form video question answering requires reasoning over extended temporal contexts, making frame selection critical for large vision-language models (LVLMs) bound by finite context windows. Existing methods face a sharp trade-off: similarity-based selectors are fast but collapse compositional queries into a single dense vector, losing sub-event ordering and cross-modal bindings; agent-based methods recover this structure through iterative LVLM inference, but at prohibitive cost. We introduce HiMu, a training-free framework that bridges this gap. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (ASR, CLAP). The resulting signals are normalized, temporally smoothed to align different modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, producing a continuous satisfaction curve. Evaluations on Video-MME, LongVideoBench and HERBench-Lite show that HiMu advances the efficiency-accuracy Pareto front: at 16 frames with Qwen3-VL 8B it outperforms all competing selectors, and with GPT-4o it surpasses agentic systems operating at 32-512 frames while requiring roughly 10x fewer FLOPs.
HiMu is a training-free framework for long-form video question answering that uses a hierarchical logic tree decomposition and lightweight experts to efficiently select relevant video frames while preserving temporal structure and cross-modal bindings.
vision-language models frame selection temporal context similarity-based selectors agent-based methods text-only LLM hierarchical logic tree atomic predicates vision experts audio experts CLIP open-vocabulary detection OCR ASR CLAP temporal smoothing fuzzy-logic operators temporal sequencing cross-modal bindings Video-MME LongVideoBench HERBench-Lite Qwen3-VL GPT-4o