Agentic Learner with Grow-and-Refine Multimodal Semantic Memory
MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo -- solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a single-modality trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both multimodal and integrated, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce ViLoMem, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge -- preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across six multimodal benchmarks, ViLoMem consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction--hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning. Our project page will be available at https://weihao-bo.github.io/ViLoMeo-page.
692918571f3fae537858b537
Weihao Bo
claimed_verified
2025-11-28T13:48:44.844Z
6927ee0fb4ba9da7891c72c9
/avatars/51875a45f74221ca79b29de5259fb0b9.svg
Weihao Bo
weihao-bo
user
weihao-bo
692918571f3fae537858b538
Shan Zhang
692918571f3fae537858b539
Yanpeng Sun
claimed_verified
2025-11-30T20:10:18.226Z
64297212e5f33939cf3a3d9b
/avatars/bd21759ab5d7e526b99fcb7ed813ffb3.svg
yanpeng_sun
syp115
user
syp115
692918571f3fae537858b53a
Jingjing Wu
692918571f3fae537858b53b
Qunyi Xie
692918571f3fae537858b53c
Xiao Tan
692918571f3fae537858b53d
Kunbin Chen
692918571f3fae537858b53e
Wei He
692918571f3fae537858b53f
Xiaofan Li
692918571f3fae537858b540
Na Zhao
692918571f3fae537858b541
Jingdong Wang
692918571f3fae537858b542
Zechao Li
Agentic Learner with Grow-and-Refine Multimodal Semantic Memory
MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo -- solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a single-modality trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both multimodal and integrated, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce ViLoMem, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge -- preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across six multimodal benchmarks, ViLoMem consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction--hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning. Our project page will be available at https://weihao-bo.github.io/ViLoMeo-page.
ViLoMem, a dual-stream memory framework, enhances MLLMs by preserving multimodal semantic knowledge, reducing errors, and improving accuracy across benchmarks.
memory-augmented agents trajectory-based memory brevity bias semantic memory multimodal problem-solving visual attention logical reasoning ViLoMem dual-stream memory schema-based memory visual distraction patterns logical reasoning errors pass@1 accuracy catastrophic forgetting lifelong learning cross-domain learning