RIVER: A Real-Time Interaction Benchmark for Video LLMs
The rapid advancement of multimodal large language models has demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering real-time interactivity. Addressing this gap, we introduce the Real-tIme Video intERaction Bench (RIVER Bench), designed for evaluating online video comprehension. RIVER Bench introduces a novel framework comprising Retrospective Memory, Live-Perception, and Proactive Anticipation tasks, closely mimicking interactive dialogues rather than responding to entire videos at once. We conducted detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online video interaction, especially their deficiencies in long-term memory and future perception, we proposed a general improvement method that enables models to interact with users more flexibly in real time. We believe this work will significantly advance the development of real-time interactive video understanding models and inspire future research in this emerging field. Datasets and code are publicly available at https://github.com/OpenGVLab/RIVER.
69a8f68b8b5b27c7ea9fa61e
Yansong Shi
claimed_verified
2026-03-05T10:01:46.025Z
64c6161b19565937fb444d5e
https://cdn-avatars.huggingface.co/v1/production/uploads/64c6161b19565937fb444d5e/Kt3kX_mdiHSr0VUmar8JJ.jpeg
Yansong Shi
nanamma
user
nanamma
69a8f68b8b5b27c7ea9fa61f
Qingsong Zhao
claimed_verified
2026-03-05T10:01:50.081Z
66b9f7c3ccd6bace75ac37f9
https://cdn-avatars.huggingface.co/v1/production/uploads/66b9f7c3ccd6bace75ac37f9/sQQtsyLBVkYzinu7zImAc.jpeg
qingsongzhao
qingsong-up
user
qingsong-up
69a8f68b8b5b27c7ea9fa620
Tianxiang Jiang
claimed_verified
2026-03-05T10:01:48.149Z
6744754ff9940208b97a6a9a
https://cdn-avatars.huggingface.co/v1/production/uploads/6744754ff9940208b97a6a9a/PRG6_0jAfsj0uoUJvKyWf.png
Tianxiang Jiang
Eurayka
user
Eurayka
69a8f68b8b5b27c7ea9fa621
Xiangyu Zeng
69a8f68b8b5b27c7ea9fa622
Yi Wang
69a8f68b8b5b27c7ea9fa623
Limin Wang
RIVER: A Real-Time Interaction Benchmark for Video LLMs
The rapid advancement of multimodal large language models has demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering real-time interactivity. Addressing this gap, we introduce the Real-tIme Video intERaction Bench (RIVER Bench), designed for evaluating online video comprehension. RIVER Bench introduces a novel framework comprising Retrospective Memory, Live-Perception, and Proactive Anticipation tasks, closely mimicking interactive dialogues rather than responding to entire videos at once. We conducted detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online video interaction, especially their deficiencies in long-term memory and future perception, we proposed a general improvement method that enables models to interact with users more flexibly in real time. We believe this work will significantly advance the development of real-time interactive video understanding models and inspire future research in this emerging field. Datasets and code are publicly available at https://github.com/OpenGVLab/RIVER.
Real-time video interaction benchmark evaluates online video comprehension capabilities through retrospective memory, live-perception, and proactive anticipation tasks, revealing limitations of offline models in real-time processing and long-term memory.
multimodal large language models offline paradigm real-time interactivity RIVER Bench Retrospective Memory Live-Perception Proactive Anticipation online video interaction long-term memory future perception