Contextualized Visual Personalization in Vision-Language Models
Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.
6982b6d89084cb4f0ecb5668
Yeongtak Oh
claimed_verified
2026-02-04T12:28:42.713Z
636f6e8a31af06da86499ebc
/avatars/9430fbc05774aad8e46c0861769b3c30.svg
Yeongtak
Yeongtak
user
Yeongtak
6982b6d89084cb4f0ecb5669
Sangwon Yu
claimed_verified
2026-02-23T09:47:29.962Z
6543504021f55472a8b99c27
/avatars/3819eed42d744d320419a5523f590143.svg
Sangwon Yu
ysw1021
user
ysw1021
6982b6d89084cb4f0ecb566a
Junsung Park
6982b6d89084cb4f0ecb566b
Han Cheol Moon
6982b6d89084cb4f0ecb566c
Jisoo Mok
6982b6d89084cb4f0ecb566d
Sungroh Yoon
Contextualized Visual Personalization in Vision-Language Models
Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.
CoViP addresses contextualized visual personalization by treating personalized image captioning as a core task and improving capabilities through reinforcement-learning-based post-training and caption-augmented generation.
vision-language models contextualized visual personalization personalized image captioning reinforcement-learning-based post-training caption-augmented generation visual recognition textual retrieval diagnostic evaluations visual context