CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answering
Knowledge-based visual question answering (KB-VQA) demonstrates significant potential for handling knowledge-intensive tasks. However, conflicts arise between static parametric knowledge in vision language models (VLMs) and dynamically retrieved information due to the static model knowledge from pre-training. The outputs either ignore retrieved contexts or exhibit inconsistent integration with parametric knowledge, posing substantial challenges for KB-VQA. Current knowledge conflict mitigation methods primarily adapted from language-based approaches, focusing on context-level conflicts through engineered prompting strategies or context-aware decoding mechanisms. However, these methods neglect the critical role of visual information in conflicts and suffer from redundant retrieved contexts, which impair accurate conflict identification and effective mitigation. To address these limitations, we propose CC-VQA: a novel training-free, conflict- and correlation-aware method for KB-VQA. Our method comprises two core components: (1) Vision-Centric Contextual Conflict Reasoning, which performs visual-semantic conflict analysis across internal and external knowledge contexts; and (2) Correlation-Guided Encoding and Decoding, featuring positional encoding compression for low-correlation statements and adaptive decoding using correlation-weighted conflict scoring. Extensive evaluations on E-VQA, InfoSeek, and OK-VQA benchmarks demonstrate that CC-VQA achieves state-of-the-art performance, yielding absolute accuracy improvements of 3.3\% to 6.4\% compared to existing methods. Code is available at https://github.com/cqu-student/CC-VQA.
69a4ef55bc2d66e68888f48a
Yuyang Hong
claimed_verified
2026-03-03T10:42:49.539Z
66e03d7123e5f162e7e5c5ac
https://cdn-avatars.huggingface.co/v1/production/uploads/66e03d7123e5f162e7e5c5ac/PHHGlQKnZhr_BNpciB9Zz.jpeg
hongyuyang
hongyuyang23casia
user
hongyuyang23casia
69a4ef55bc2d66e68888f48b
Jiaqi Gu
69a4ef55bc2d66e68888f48c
Yujin Lou
69a4ef55bc2d66e68888f48d
Lubin Fan
69a4ef55bc2d66e68888f48e
Qi Yang
69a4ef55bc2d66e68888f48f
Ying Wang
69a4ef55bc2d66e68888f490
Kun Ding
69a4ef55bc2d66e68888f491
Yue Wu
69a4ef55bc2d66e68888f492
Shiming Xiang
69a4ef55bc2d66e68888f493
Jieping Ye
CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answering
Knowledge-based visual question answering (KB-VQA) demonstrates significant potential for handling knowledge-intensive tasks. However, conflicts arise between static parametric knowledge in vision language models (VLMs) and dynamically retrieved information due to the static model knowledge from pre-training. The outputs either ignore retrieved contexts or exhibit inconsistent integration with parametric knowledge, posing substantial challenges for KB-VQA. Current knowledge conflict mitigation methods primarily adapted from language-based approaches, focusing on context-level conflicts through engineered prompting strategies or context-aware decoding mechanisms. However, these methods neglect the critical role of visual information in conflicts and suffer from redundant retrieved contexts, which impair accurate conflict identification and effective mitigation. To address these limitations, we propose CC-VQA: a novel training-free, conflict- and correlation-aware method for KB-VQA. Our method comprises two core components: (1) Vision-Centric Contextual Conflict Reasoning, which performs visual-semantic conflict analysis across internal and external knowledge contexts; and (2) Correlation-Guided Encoding and Decoding, featuring positional encoding compression for low-correlation statements and adaptive decoding using correlation-weighted conflict scoring. Extensive evaluations on E-VQA, InfoSeek, and OK-VQA benchmarks demonstrate that CC-VQA achieves state-of-the-art performance, yielding absolute accuracy improvements of 3.3\% to 6.4\% compared to existing methods. Code is available at https://github.com/cqu-student/CC-VQA.
CC-VQA addresses knowledge conflicts in visual question answering by incorporating visual-semantic conflict analysis and correlation-guided encoding-decoding mechanisms without requiring model retraining.
knowledge-based visual question answering vision language models parametric knowledge dynamic retrieval context-level conflicts engineered prompting context-aware decoding visual-semantic conflict analysis correlation-guided encoding adaptive decoding positional encoding compression