EgoAVU: Egocentric Audio-Visual Understanding
Understanding egocentric videos plays a vital role for embodied intelligence. Recent multi-modal large language models (MLLMs) can accept both visual and audio inputs. However, due to the challenge of obtaining text labels with coherent joint-modality information, whether MLLMs can jointly understand both modalities in egocentric videos remains under-explored. To address this problem, we introduce EgoAVU, a scalable data engine to automatically generate egocentric audio-visual narrations, questions, and answers. EgoAVU enriches human narrations with multimodal context and generates audio-visual narrations through cross-modal correlation modeling. Token-based video filtering and modular, graph-based curation ensure both data diversity and quality. Leveraging EgoAVU, we construct EgoAVU-Instruct, a large-scale training dataset of 3M samples, and EgoAVU-Bench, a manually verified evaluation split covering diverse tasks. EgoAVU-Bench clearly reveals the limitations of existing MLLMs: they bias heavily toward visual signals, often neglecting audio cues or failing to correspond audio with the visual source. Finetuning MLLMs on EgoAVU-Instruct effectively addresses this issue, enabling up to 113% performance improvement on EgoAVU-Bench. Such benefits also transfer to other benchmarks such as EgoTempo and EgoIllusion, achieving up to 28% relative performance gain. Code will be released to the community.
69894d1dbeecc443208d25fb
Ashish Seth
69894d1dbeecc443208d25fc
Xinhao Mei
69894d1dbeecc443208d25fd
Changsheng Zhao
69894d1dbeecc443208d25fe
Varun Nagaraja
69894d1dbeecc443208d25ff
Ernie Chang
69894d1dbeecc443208d2600
Gregory P. Meyer
69894d1dbeecc443208d2601
Gael Le Lan
69894d1dbeecc443208d2602
Yunyang Xiong
69894d1dbeecc443208d2603
Vikas Chandra
69894d1dbeecc443208d2604
Yangyang Shi
69894d1dbeecc443208d2605
Dinesh Manocha
69894d1dbeecc443208d2606
Zhipeng Cai
EgoAVU: Egocentric Audio-Visual Understanding
Understanding egocentric videos plays a vital role for embodied intelligence. Recent multi-modal large language models (MLLMs) can accept both visual and audio inputs. However, due to the challenge of obtaining text labels with coherent joint-modality information, whether MLLMs can jointly understand both modalities in egocentric videos remains under-explored. To address this problem, we introduce EgoAVU, a scalable data engine to automatically generate egocentric audio-visual narrations, questions, and answers. EgoAVU enriches human narrations with multimodal context and generates audio-visual narrations through cross-modal correlation modeling. Token-based video filtering and modular, graph-based curation ensure both data diversity and quality. Leveraging EgoAVU, we construct EgoAVU-Instruct, a large-scale training dataset of 3M samples, and EgoAVU-Bench, a manually verified evaluation split covering diverse tasks. EgoAVU-Bench clearly reveals the limitations of existing MLLMs: they bias heavily toward visual signals, often neglecting audio cues or failing to correspond audio with the visual source. Finetuning MLLMs on EgoAVU-Instruct effectively addresses this issue, enabling up to 113% performance improvement on EgoAVU-Bench. Such benefits also transfer to other benchmarks such as EgoTempo and EgoIllusion, achieving up to 28% relative performance gain. Code will be released to the community.
Multi-modal large language models struggle to jointly understand audio and visual signals in egocentric videos, but a new scalable data engine and dataset significantly improve their performance through targeted fine-tuning.
multi-modal large language models egocentric videos audio-visual narrations cross-modal correlation modeling token-based video filtering modular curation graph-based curation EgoAVU-Instruct EgoAVU-Bench EgoTempo EgoIllusion