AURA: Always-On Understanding and Real-Time Assistance via Video Streams
Video Large Language Models (VideoLLMs) have achieved strong performance on many video understanding tasks, but most existing systems remain offline and are not well-suited for live video streams that require continuous observation and timely response. Recent streaming VideoLLMs have made progress, yet current approaches often rely on decoupled trigger-response pipelines or are limited to captioning-style narration, reducing their effectiveness for open-ended question answering and long-horizon interaction. We propose AURA (Always-On Understanding and Real-Time Assistance), an end-to-end streaming visual interaction framework that enables a unified VideoLLM to continuously process video streams and support both real-time question answering and proactive responses. AURA integrates context management, data construction, training objectives, and deployment optimization for stable long-horizon streaming interaction. It achieves state-of-the-art performance on streaming benchmarks and supports a real-time demo system with ASR and TTS running at 2 FPS on two 80G accelerators. We release the AURA model together with a real-time inference framework to facilitate future research.
69d46b222c523d36431fb067
Xudong Lu
claimed_verified
2026-04-07T08:34:41.662Z
642e686bbe01b88c9446db8b
https://cdn-avatars.huggingface.co/v1/production/uploads/642e686bbe01b88c9446db8b/tb1DKe5xt50ykOeXiUuTE.jpeg
Lu Xudong
lucky-lance
user
lucky-lance
69d46b222c523d36431fb068
Yang Bo
69d46b222c523d36431fb069
Jinpeng Chen
claimed_verified
2026-04-13T07:34:59.888Z
669a02fa61278f96d87902f5
https://cdn-avatars.huggingface.co/v1/production/uploads/669a02fa61278f96d87902f5/UsqqKUizH2w0mlBk0z5xK.jpeg
Jinpeng Chen
jinpeng0528
user
jinpeng0528
69d46b222c523d36431fb06a
Shuhan Li
69d46b222c523d36431fb06b
Xintong Guo
69d46b222c523d36431fb06c
Huankang Guan
69d46b222c523d36431fb06d
Fang Liu
69d46b222c523d36431fb06e
Dunyuan Xu
69d46b222c523d36431fb06f
Peiwen Sun
69d46b222c523d36431fb070
Heyang Sun
69d46b222c523d36431fb071
Rui Liu
69d46b222c523d36431fb072
Hongsheng Li
AURA: Always-On Understanding and Real-Time Assistance via Video Streams
Video Large Language Models (VideoLLMs) have achieved strong performance on many video understanding tasks, but most existing systems remain offline and are not well-suited for live video streams that require continuous observation and timely response. Recent streaming VideoLLMs have made progress, yet current approaches often rely on decoupled trigger-response pipelines or are limited to captioning-style narration, reducing their effectiveness for open-ended question answering and long-horizon interaction. We propose AURA (Always-On Understanding and Real-Time Assistance), an end-to-end streaming visual interaction framework that enables a unified VideoLLM to continuously process video streams and support both real-time question answering and proactive responses. AURA integrates context management, data construction, training objectives, and deployment optimization for stable long-horizon streaming interaction. It achieves state-of-the-art performance on streaming benchmarks and supports a real-time demo system with ASR and TTS running at 2 FPS on two 80G accelerators. We release the AURA model together with a real-time inference framework to facilitate future research.
AURA is an end-to-end streaming visual interaction framework that enables continuous video stream processing with real-time question answering and proactive responses through integrated context management and optimized deployment.
VideoLLMs streaming VideoLLMs real-time question answering proactive responses context management data construction training objectives deployment optimization long-horizon streaming interaction ASR TTS