Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline
While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.
69aa4a7bc4f6edddcc6c8668
Guo Chen
69aa4a7bc4f6edddcc6c8669
Lidong Lu
claimed_verified
2026-03-06T13:55:24.638Z
64a3de701698ad2985277148
/avatars/09eebadbbea53ed2800591564ff5c931.svg
lulidong
lulidong
user
lulidong
69aa4a7bc4f6edddcc6c866a
Yicheng Liu
69aa4a7bc4f6edddcc6c866b
Liangrui Dong
69aa4a7bc4f6edddcc6c866c
Lidong Zou
69aa4a7bc4f6edddcc6c866d
Jixin Lv
69aa4a7bc4f6edddcc6c866e
Zhenquan Li
69aa4a7bc4f6edddcc6c866f
Xinyi Mao
69aa4a7bc4f6edddcc6c8670
Baoqi Pei
69aa4a7bc4f6edddcc6c8671
Shihao Wang
69aa4a7bc4f6edddcc6c8672
Zhiqi Li
69aa4a7bc4f6edddcc6c8673
Karan Sapra
69aa4a7bc4f6edddcc6c8674
Fuxiao Liu
69aa4a7bc4f6edddcc6c8675
Yin-Dong Zheng
69aa4a7bc4f6edddcc6c8676
Yifei Huang
69aa4a7bc4f6edddcc6c8677
Limin Wang
69aa4a7bc4f6edddcc6c8678
Zhiding Yu
69aa4a7bc4f6edddcc6c8679
Andrew Tao
69aa4a7bc4f6edddcc6c867a
Guilin Liu
69aa4a7bc4f6edddcc6c867b
Tong Lu
Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline
While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.
MM-Lifelong dataset captures natural video sequences across multiple temporal scales to evaluate multimodal lifelong understanding, revealing limitations in current approaches and introducing a recursive agent framework for improved performance.
multimodal lifelong understanding MLLMs working memory bottleneck global localization collapse recursive multimodal agent dynamic memory management recursive belief state