VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.8% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.
6923e4d1b5612535ed9559de
Xinlei Yu
6923e4d1b5612535ed9559df
Chengming Xu
6923e4d1b5612535ed9559e0
Guibin Zhang
6923e4d1b5612535ed9559e1
Zhangquan Chen
6923e4d1b5612535ed9559e2
Yudong Zhang
6923e4d1b5612535ed9559e3
Yongbo He
6923e4d1b5612535ed9559e4
Peng-Tao Jiang
claimed_verified
2025-11-27T10:00:32.772Z
6423efbdb77cc3daf8429755
/avatars/a5b480713b4dd1dae8191545cb4c6f94.svg
Peng-Tao Jiang
ptjiang
user
ptjiang
6923e4d1b5612535ed9559e5
Jiangning Zhang
6923e4d1b5612535ed9559e6
Xiaobin Hu
6923e4d1b5612535ed9559e7
Shuicheng Yan
VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.8% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.
VisMem enhances Vision-Language Models by incorporating dynamic latent vision memories, improving performance on complex visual tasks through perceptual fidelity and semantic consistency.
Visual-Language Models visual processing bottleneck short-term visually-dominant memory long-term semantically-dominant memory dynamic latent vision memories perceptual retention semantic consolidation visual understanding visual reasoning visual generation latent-space memory enhancement