Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VLMs against cross-modal adversarial attacks, and propose Function-word De-Attention (FDA) accordingly to mitigate the impact of function words. Similar to differential amplifiers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more aligned and robust VLMs. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and 3 models. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/0.6% performance drops on the 3 tested models on retrieval, and a 90% ASR drop with a 0.3% performance gain on visual grounding. We demonstrate the scalability, generalization, and zero-shot performance of FDA experimentally, as well as in-depth ablation studies and analysis. Code will be made publicly at https://github.com/michaeltian108/FDA.
693a41e374fced5bf9c32477
Qiwei Tian
admin_assigned
2025-12-11T10:54:08.007Z
689b16601133f93a6f4afdca
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/MWlw4vtHV6ptOpTT6-U57.png
Qiwei Tian
michaeltqw108
user
michaeltqw108
693a41e374fced5bf9c32478
Chenhao Lin
admin_assigned
2025-12-11T10:54:14.957Z
669389688db712bad7c70e44
/avatars/1938192a68bd231a39c662e379cfbb7f.svg
CHEN HAO LIN
chl810903
user
chl810903
693a41e374fced5bf9c32479
Zhengyu Zhao
admin_assigned
2025-12-11T10:54:20.650Z
634cc244a5d4e109c6af9375
/avatars/ebc7e2c966b7434f26def3f05da45cd9.svg
zhengyuzhao
zhaozhengyu
user
zhaozhengyu
693a41e374fced5bf9c3247a
Chao Shen
Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VLMs against cross-modal adversarial attacks, and propose Function-word De-Attention (FDA) accordingly to mitigate the impact of function words. Similar to differential amplifiers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more aligned and robust VLMs. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and 3 models. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/0.6% performance drops on the 3 tested models on retrieval, and a 90% ASR drop with a 0.3% performance gain on visual grounding. We demonstrate the scalability, generalization, and zero-shot performance of FDA experimentally, as well as in-depth ablation studies and analysis. Code will be made publicly at https://github.com/michaeltian108/FDA.
Function-word De-Attention (FDA) mitigates adversarial attacks on robust VLMs by differentially subtracting function-word cross-attention, improving robustness with minimal performance trade-offs.
function words cross-modal adversarial attacks Function-word De-Attention FDA differential amplifiers cross-attention VLMs ASR retrieval visual grounding