StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications like AR glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether MLLMs can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively evaluate streaming video understanding. These tasks assess whether models can use real-time gaze to follow shifting attention and infer user intentions from only past and currently observed frames. To build StreamGaze, we develop a gaze-video QA generation pipeline that aligns egocentric videos with raw gaze trajectories via fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that closely reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, revealing fundamental limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze-prompting strategies, reasoning behaviors, and task-specific failure modes, offering deeper insight into why current MLLMs struggle and what capabilities future models must develop. All data and code will be publicly released to support continued research in gaze-guided streaming video understanding.
692e822337312eaa83fd88ef
Daeun Lee
claimed_verified
2026-03-16T10:46:30.422Z
647585665aba8edfb2f393d7
https://cdn-avatars.huggingface.co/v1/production/uploads/647585665aba8edfb2f393d7/LU4xZwZcSoU6VcddKE2Eg.jpeg
Daeun Lee
daeunni
user
daeunni
692e822337312eaa83fd88f0
Subhojyoti Mukherjee
claimed_verified
2026-03-16T10:46:32.368Z
69a8b582edaa208c498deee0
/avatars/7b17e67fa2bd4f4c4a1aa72ecffd2858.svg
Subhojyoti Mukherjee
subhojyoti1990
user
subhojyoti1990
692e822337312eaa83fd88f1
Branislav Kveton
692e822337312eaa83fd88f2
Ryan A. Rossi
692e822337312eaa83fd88f3
Viet Dac Lai
692e822337312eaa83fd88f4
Seunghyun Yoon
692e822337312eaa83fd88f5
Trung Bui
692e822337312eaa83fd88f6
Franck Dernoncourt
claimed_verified
2026-01-26T08:34:32.332Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
Franck-Dernoncourt
user
Franck-Dernoncourt
692e822337312eaa83fd88f7
Mohit Bansal
StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications like AR glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether MLLMs can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively evaluate streaming video understanding. These tasks assess whether models can use real-time gaze to follow shifting attention and infer user intentions from only past and currently observed frames. To build StreamGaze, we develop a gaze-video QA generation pipeline that aligns egocentric videos with raw gaze trajectories via fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that closely reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, revealing fundamental limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze-prompting strategies, reasoning behaviors, and task-specific failure modes, offering deeper insight into why current MLLMs struggle and what capabilities future models must develop. All data and code will be publicly released to support continued research in gaze-guided streaming video understanding.
StreamGaze is a benchmark that evaluates how effectively models use gaze signals for temporal and proactive reasoning in streaming videos, revealing limitations in current MLLM capabilities.
MLLMs temporal reasoning gaze signals gaze-guided tasks past present proactive fixation extraction region-specific visual prompting scanpath construction spatio-temporally grounded QA human perceptual dynamics gaze-prompting strategies reasoning behaviors failure modes