OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation
Recent advances in multi-modal large language models (MLLMs) have enabled unified perception-reasoning capabilities, yet these systems remain highly vulnerable to jailbreak attacks that bypass safety alignment and induce harmful behaviors. Existing benchmarks such as JailBreakV-28K, MM-SafetyBench, and HADES provide valuable insights into multi-modal vulnerabilities, but they typically focus on limited attack scenarios, lack standardized defense evaluation, and offer no unified, reproducible toolbox. To address these gaps, we introduce OmniSafeBench-MM, which is a comprehensive toolbox for multi-modal jailbreak attack-defense evaluation. OmniSafeBench-MM integrates 13 representative attack methods, 15 defense strategies, and a diverse dataset spanning 9 major risk domains and 50 fine-grained categories, structured across consultative, imperative, and declarative inquiry types to reflect realistic user intentions. Beyond data coverage, it establishes a three-dimensional evaluation protocol measuring (1) harmfulness, distinguished by a granular, multi-level scale ranging from low-impact individual harm to catastrophic societal threats, (2) intent alignment between responses and queries, and (3) response detail level, enabling nuanced safety-utility analysis. We conduct extensive experiments on 10 open-source and 8 closed-source MLLMs to reveal their vulnerability to multi-modal jailbreak. By unifying data, methodology, and evaluation into an open-source, reproducible platform, OmniSafeBench-MM provides a standardized foundation for future research. The code is released at https://github.com/jiaxiaojunQAQ/OmniSafeBench-MM.
693811de19d912300c34a57d
Xiaojun Jia
claimed_verified
2025-12-09T13:52:46.579Z
64c6627d5671d42e0adfad56
/avatars/8b98054b2911b86dcc4856a15306e60f.svg
jiaxiaojunQAQ
jiaxiaojunQAQ
user
jiaxiaojunQAQ
693811de19d912300c34a57e
Jie Liao
claimed_verified
2025-12-10T08:56:56.754Z
659e36dc18dc736029331d49
/avatars/97f03036b0bd647d3b9dc5839db07d21.svg
Liao, Jie
Jeff0481
user
Jeff0481
693811de19d912300c34a57f
Qi Guo
693811de19d912300c34a580
Teng Ma
693811de19d912300c34a581
Simeng Qin
693811de19d912300c34a582
Ranjie Duan
693811de19d912300c34a583
Tianlin Li
693811de19d912300c34a584
Yihao Huang
693811de19d912300c34a585
Zhitao Zeng
693811de19d912300c34a586
Dongxian Wu
693811de19d912300c34a587
Yiming Li
693811de19d912300c34a588
Wenqi Ren
693811de19d912300c34a589
Xiaochun Cao
693811de19d912300c34a58a
Yang Liu
OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation
Recent advances in multi-modal large language models (MLLMs) have enabled unified perception-reasoning capabilities, yet these systems remain highly vulnerable to jailbreak attacks that bypass safety alignment and induce harmful behaviors. Existing benchmarks such as JailBreakV-28K, MM-SafetyBench, and HADES provide valuable insights into multi-modal vulnerabilities, but they typically focus on limited attack scenarios, lack standardized defense evaluation, and offer no unified, reproducible toolbox. To address these gaps, we introduce OmniSafeBench-MM, which is a comprehensive toolbox for multi-modal jailbreak attack-defense evaluation. OmniSafeBench-MM integrates 13 representative attack methods, 15 defense strategies, and a diverse dataset spanning 9 major risk domains and 50 fine-grained categories, structured across consultative, imperative, and declarative inquiry types to reflect realistic user intentions. Beyond data coverage, it establishes a three-dimensional evaluation protocol measuring (1) harmfulness, distinguished by a granular, multi-level scale ranging from low-impact individual harm to catastrophic societal threats, (2) intent alignment between responses and queries, and (3) response detail level, enabling nuanced safety-utility analysis. We conduct extensive experiments on 10 open-source and 8 closed-source MLLMs to reveal their vulnerability to multi-modal jailbreak. By unifying data, methodology, and evaluation into an open-source, reproducible platform, OmniSafeBench-MM provides a standardized foundation for future research. The code is released at https://github.com/jiaxiaojunQAQ/OmniSafeBench-MM.
OmniSafeBench-MM is a comprehensive tool for evaluating multi-modal jailbreak attacks and defenses, covering various attack methods, defense strategies, and risk domains.
multi-modal large language models jailbreak attacks safety alignment OmniSafeBench-MM attack methods defense strategies risk domains harmfulness intent alignment response detail level