Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
Video Large Language Models (VideoLLMs) extend the capabilities of
vision-language models to spatiotemporal inputs, enabling tasks such as video
question answering (VideoQA). Despite recent advances in VideoLLMs, their
internal mechanisms on where and how they extract and propagate video and
textual information remain less explored. In this study, we investigate the
internal information flow of VideoLLMs using mechanistic interpretability
techniques. Our analysis reveals consistent patterns across diverse VideoQA
tasks: (1) temporal reasoning in VideoLLMs initiates with active cross-frame
interactions in early-to-middle layers, (2) followed by progressive
video-language integration in middle layers. This is facilitated by alignment
between video representations and linguistic embeddings containing temporal
concepts. (3) Upon completion of this integration, the model is ready to
generate correct answers in middle-to-late layers. (4) Based on our analysis,
we show that VideoLLMs can retain their VideoQA performance by selecting these
effective information pathways while suppressing a substantial amount of
attention edges, e.g., 58% in LLaVA-NeXT-7B-Video-FT. These findings provide a
blueprint on how VideoLLMs perform temporal reasoning and offer practical
insights for improving model interpretability and downstream generalization.
Our project page with the source code is available at
https://map-the-flow.github.io
68f1a9556e0bef323a68fd7e
Minji Kim
claimed_verified
2025-10-23T15:02:00.655Z
66e345c9596fcff3e4b22e5a
https://cdn-avatars.huggingface.co/v1/production/uploads/66e345c9596fcff3e4b22e5a/tWZy_N9f6--au4T0utEpZ.jpeg
Minji Kim
byminji
user
byminji
68f1a9556e0bef323a68fd7f
Taekyung Kim
claimed_verified
2025-10-17T04:10:22.216Z
67c6a1e75e2443d7d5f85cb3
/avatars/0569b368520411ab828d46725bc3896a.svg
Taekyung Kim
taekyung-k
user
taekyung-k
68f1a9556e0bef323a68fd80
Bohyung Han
Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
Video Large Language Models (VideoLLMs) extend the capabilities of
vision-language models to spatiotemporal inputs, enabling tasks such as video
question answering (VideoQA). Despite recent advances in VideoLLMs, their
internal mechanisms on where and how they extract and propagate video and
textual information remain less explored. In this study, we investigate the
internal information flow of VideoLLMs using mechanistic interpretability
techniques. Our analysis reveals consistent patterns across diverse VideoQA
tasks: (1) temporal reasoning in VideoLLMs initiates with active cross-frame
interactions in early-to-middle layers, (2) followed by progressive
video-language integration in middle layers. This is facilitated by alignment
between video representations and linguistic embeddings containing temporal
concepts. (3) Upon completion of this integration, the model is ready to
generate correct answers in middle-to-late layers. (4) Based on our analysis,
we show that VideoLLMs can retain their VideoQA performance by selecting these
effective information pathways while suppressing a substantial amount of
attention edges, e.g., 58% in LLaVA-NeXT-7B-Video-FT. These findings provide a
blueprint on how VideoLLMs perform temporal reasoning and offer practical
insights for improving model interpretability and downstream generalization.
Our project page with the source code is available at
https://map-the-flow.github.io
Video Large Language Models (VideoLLMs) perform video question answering by initiating temporal reasoning through cross-frame interactions, followed by video-language integration, and generate answers using effective information pathways while suppressing unnecessary attention edges.
Video Large Language Models VideoLLMs video question answering VideoQA mechanistic interpretability temporal reasoning cross-frame interactions video-language integration linguistic embeddings attention edges