Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
Recent advances in Vision-Language Models (VLMs) have achieved
state-of-the-art performance on numerous benchmark tasks. However, the use of
internet-scale, often proprietary, pretraining corpora raises a critical
concern for both practitioners and users: inflated performance due to test-set
leakage. While prior works have proposed mitigation strategies such as
decontamination of pretraining data and benchmark redesign for LLMs, the
complementary direction of developing detection methods for contaminated VLMs
remains underexplored. To address this gap, we deliberately contaminate
open-source VLMs on popular benchmarks and show that existing detection
approaches either fail outright or exhibit inconsistent behavior. We then
propose a novel simple yet effective detection method based on multi-modal
semantic perturbation, demonstrating that contaminated models fail to
generalize under controlled perturbations. Finally, we validate our approach
across multiple realistic contamination strategies, confirming its robustness
and effectiveness. The code and perturbed dataset will be released publicly.
690d8af0ad2597bf6c464f4f
Jaden Park
claimed_verified
2025-11-07T10:28:27.216Z
670f5267d1b58394145c1ca3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/t-YchgvZCbDW-plR8DZbA.png
Jaden Park
jpark677
user
jpark677
690d8af0ad2597bf6c464f50
Mu Cai
690d8af0ad2597bf6c464f51
Feng Yao
claimed_verified
2025-11-10T09:31:28.714Z
64c8b2c5c547ed5243e14a6e
/avatars/96d4a9010f96001c8cff235915926390.svg
Feng Yao
fengyao1909
user
fengyao1909
690d8af0ad2597bf6c464f52
Jingbo Shang
690d8af0ad2597bf6c464f53
Soochahn Lee
690d8af0ad2597bf6c464f54
Yong Jae Lee
Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
Recent advances in Vision-Language Models (VLMs) have achieved
state-of-the-art performance on numerous benchmark tasks. However, the use of
internet-scale, often proprietary, pretraining corpora raises a critical
concern for both practitioners and users: inflated performance due to test-set
leakage. While prior works have proposed mitigation strategies such as
decontamination of pretraining data and benchmark redesign for LLMs, the
complementary direction of developing detection methods for contaminated VLMs
remains underexplored. To address this gap, we deliberately contaminate
open-source VLMs on popular benchmarks and show that existing detection
approaches either fail outright or exhibit inconsistent behavior. We then
propose a novel simple yet effective detection method based on multi-modal
semantic perturbation, demonstrating that contaminated models fail to
generalize under controlled perturbations. Finally, we validate our approach
across multiple realistic contamination strategies, confirming its robustness
and effectiveness. The code and perturbed dataset will be released publicly.
A novel detection method based on multi-modal semantic perturbation is proposed to identify contaminated Vision-Language Models, demonstrating robustness across various contamination strategies.
Vision-Language Models VLMs pretraining corpora test-set leakage decontamination benchmark redesign detection methods multi-modal semantic perturbation