The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoning
As Multimodal Large Language Models (MLLMs) acquire stronger reasoning capabilities to handle complex, multi-image instructions, this advancement may pose new safety risks. We study this problem by introducing MIR-SafetyBench, the first benchmark focused on multi-image reasoning safety, which consists of 2,676 instances across a taxonomy of 9 multi-image relations. Our extensive evaluations on 19 MLLMs reveal a troubling trend: models with more advanced multi-image reasoning can be more vulnerable on MIR-SafetyBench. Beyond attack success rates, we find that many responses labeled as safe are superficial, often driven by misunderstanding or evasive, non-committal replies. We further observe that unsafe generations exhibit lower attention entropy than safe ones on average. This internal signature suggests a possible risk that models may over-focus on task solving while neglecting safety constraints. Our code and data are available at https://github.com/thu-coai/MIR-SafetyBench.
69721234c1c7409747bf9799
Renmiao Chen
claimed_verified
2026-01-23T09:38:50.162Z
66f91b549e8e332d731f2e7a
/avatars/4bfe8b0a1ece93d3c745a0b288d8c55f.svg
Renmiao Chen
smilestruggler
user
smilestruggler
69721234c1c7409747bf979a
Yida Lu
69721234c1c7409747bf979b
Shiyao Cui
admin_assigned
2026-01-27T14:42:34.438Z
6617a9c192abaae4ecd5c066
/avatars/64994d014fac7f84e2925e568cc047f8.svg
Shiyao Cui
ShiyaoCui
user
ShiyaoCui
69721234c1c7409747bf979c
Xuan Ouyang
admin_assigned
2026-01-27T14:42:11.991Z
67e617d4470f96a302734e16
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QHrYmNlTRxKR1KRS50pkf.png
Xuan Ouyang
YoungXuan
user
YoungXuan
69721234c1c7409747bf979d
Victor Shea-Jay Huang
admin_assigned
2026-01-27T14:42:06.787Z
65b5d65b625ac670a79b52a8
https://cdn-avatars.huggingface.co/v1/production/uploads/65b5d65b625ac670a79b52a8/aBPXeSgh-_HQQWuPN89OK.jpeg
Victor Shea-Jay Huang
jeix
user
jeix
69721234c1c7409747bf979e
Shumin Zhang
admin_assigned
2026-01-27T14:42:17.270Z
664694c9b6c9eb0638bd27a9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dfDxBdN9zOITbGyumGJ8z.png
Shumin Zhang
shuminCA
user
shuminCA
69721234c1c7409747bf979f
Chengwei Pan
69721234c1c7409747bf97a0
Han Qiu
69721234c1c7409747bf97a1
Minlie Huang
The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoning
As Multimodal Large Language Models (MLLMs) acquire stronger reasoning capabilities to handle complex, multi-image instructions, this advancement may pose new safety risks. We study this problem by introducing MIR-SafetyBench, the first benchmark focused on multi-image reasoning safety, which consists of 2,676 instances across a taxonomy of 9 multi-image relations. Our extensive evaluations on 19 MLLMs reveal a troubling trend: models with more advanced multi-image reasoning can be more vulnerable on MIR-SafetyBench. Beyond attack success rates, we find that many responses labeled as safe are superficial, often driven by misunderstanding or evasive, non-committal replies. We further observe that unsafe generations exhibit lower attention entropy than safe ones on average. This internal signature suggests a possible risk that models may over-focus on task solving while neglecting safety constraints. Our code and data are available at https://github.com/thu-coai/MIR-SafetyBench.
Multi-image reasoning safety benchmark reveals that advanced multimodal models exhibit increased vulnerability and superficial safety responses, with unsafe generations showing lower attention entropy.
Multimodal Large Language Models multi-image reasoning safety benchmark attention entropy