Few Tokens Matter: Entropy Guided Attacks on Vision-Language Models
Vision-language models (VLMs) achieve remarkable performance but remain vulnerable to adversarial attacks. Entropy, a measure of model uncertainty, is strongly correlated with the reliability of VLM. Prior entropy-based attacks maximize uncertainty at all decoding steps, implicitly assuming that every token contributes equally to generation instability. We show instead that a small fraction (about 20%) of high-entropy tokens, i.e., critical decision points in autoregressive generation, disproportionately governs output trajectories. By concentrating adversarial perturbations on these positions, we achieve semantic degradation comparable to global methods while using substantially smaller budgets. More importantly, across multiple representative VLMs, such selective attacks convert 35-49% of benign outputs into harmful ones, exposing a more critical safety risk. Remarkably, these vulnerable high-entropy forks recur across architecturally diverse VLMs, enabling feasible transferability (17-26% harmful rates on unseen targets). Motivated by these findings, we propose Entropy-bank Guided Adversarial attacks (EGA), which achieves competitive attack success rates (93-95%) alongside high harmful conversion, thereby revealing new weaknesses in current VLM safety mechanisms.
6960e5825b7998385e6396da
Mengqi He
6960e5825b7998385e6396db
Xinyu Tian
6960e5825b7998385e6396dc
Xin Shen
6960e5825b7998385e6396dd
Jinhong Ni
admin_assigned
2026-01-09T15:57:14.555Z
64c71a5647418a0a59e5c7cb
/avatars/a99ab24c0c19b1399d2e6795fb9d7000.svg
Jinhong Ni
mcleanie
user
mcleanie
6960e5825b7998385e6396de
Shu Zou
6960e5825b7998385e6396df
Zhaoyuan Yang
admin_assigned
2026-01-09T15:57:08.195Z
6364187d969bdae89e12b209
/avatars/290793f85e8dc2ae0cd4fe1becb4b1d0.svg
zhaoyuan yang
zhaoyuan
user
zhaoyuan
6960e5825b7998385e6396e0
Jing Zhang
Few Tokens Matter: Entropy Guided Attacks on Vision-Language Models
Vision-language models (VLMs) achieve remarkable performance but remain vulnerable to adversarial attacks. Entropy, a measure of model uncertainty, is strongly correlated with the reliability of VLM. Prior entropy-based attacks maximize uncertainty at all decoding steps, implicitly assuming that every token contributes equally to generation instability. We show instead that a small fraction (about 20%) of high-entropy tokens, i.e., critical decision points in autoregressive generation, disproportionately governs output trajectories. By concentrating adversarial perturbations on these positions, we achieve semantic degradation comparable to global methods while using substantially smaller budgets. More importantly, across multiple representative VLMs, such selective attacks convert 35-49% of benign outputs into harmful ones, exposing a more critical safety risk. Remarkably, these vulnerable high-entropy forks recur across architecturally diverse VLMs, enabling feasible transferability (17-26% harmful rates on unseen targets). Motivated by these findings, we propose Entropy-bank Guided Adversarial attacks (EGA), which achieves competitive attack success rates (93-95%) alongside high harmful conversion, thereby revealing new weaknesses in current VLM safety mechanisms.
Selective adversarial attacks targeting high-entropy tokens in vision-language models achieve significant semantic degradation with reduced budgets and demonstrate transferable vulnerabilities across different architectures.
vision-language models entropy adversarial attacks autoregressive generation high-entropy tokens semantic degradation attack success rates transferability