Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment
Recent advancement in multimodal LLMs (MLLMs) has demonstrated their remarkable capability to generate descriptive captions for input videos. However, these models suffer from factual inaccuracies in the generated descriptions, causing severe hallucination issues. While prior works have explored alleviating hallucinations for static images, jointly mitigating visual object and temporal action hallucinations for dynamic videos remains a challenging and unsolved task. To tackle this challenge, we propose a Self-Augmented Contrastive Alignment (SANTA) framework for enabling object and action faithfulness by exempting the spurious correlations and enforcing the emphasis on visual facts. SANTA employs a hallucinative self-augmentation scheme to identify the potential hallucinations that lie in the MLLM and transform the original captions to the contrasted negatives. Furthermore, we develop a tracklet-phrase contrastive alignment to match the regional objects and relation-guided actions with their corresponding visual and temporal phrases. Extensive experiments demonstrate that SANTA outperforms existing methods in alleviating object and action hallucinations, yielding superior performance on the hallucination examination benchmarks.
693242b86d1060ca587a2607
Kai-Po Chang
claimed_verified
2025-12-05T08:29:53.800Z
65dad90a80bafdfb4bc94dc4
/avatars/7131b82e11245744760e8a45ba914161.svg
Kai-Po Chang
kaipochang0810
user
kaipochang0810
693242b86d1060ca587a2608
Wei-Yuan Cheng
693242b86d1060ca587a2609
Chi-Pin Huang
693242b86d1060ca587a260a
Fu-En Yang
693242b86d1060ca587a260b
Yu-Chiang Frank Wang
Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment
Recent advancement in multimodal LLMs (MLLMs) has demonstrated their remarkable capability to generate descriptive captions for input videos. However, these models suffer from factual inaccuracies in the generated descriptions, causing severe hallucination issues. While prior works have explored alleviating hallucinations for static images, jointly mitigating visual object and temporal action hallucinations for dynamic videos remains a challenging and unsolved task. To tackle this challenge, we propose a Self-Augmented Contrastive Alignment (SANTA) framework for enabling object and action faithfulness by exempting the spurious correlations and enforcing the emphasis on visual facts. SANTA employs a hallucinative self-augmentation scheme to identify the potential hallucinations that lie in the MLLM and transform the original captions to the contrasted negatives. Furthermore, we develop a tracklet-phrase contrastive alignment to match the regional objects and relation-guided actions with their corresponding visual and temporal phrases. Extensive experiments demonstrate that SANTA outperforms existing methods in alleviating object and action hallucinations, yielding superior performance on the hallucination examination benchmarks.
The SANTA framework addresses hallucinations in multimodal LLMs by using self-augmented contrastive alignment to enhance object and action faithfulness in video caption generation.
multimodal LLMs MLLMs descriptive captions factual inaccuracies hallucination issues self-augmented contrastive alignment visual object temporal action spurious correlations visual facts hallucinative self-augmentation contrasted negatives tracklet-phrase contrastive alignment regional objects relation-guided actions visual phrases temporal phrases hallucination examination benchmarks