OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation. To address this, we introduce OpenRT, a unified, modular, and high-throughput red-teaming framework designed for comprehensive MLLM safety evaluation. At its core, OpenRT architects a paradigm shift in automated red-teaming by introducing an adversarial kernel that enables modular separation across five critical dimensions: model integration, dataset management, attack strategies, judging methods, and evaluation metrics. By standardizing attack interfaces, it decouples adversarial logic from a high-throughput asynchronous runtime, enabling systematic scaling across diverse models. Our framework integrates 37 diverse attack methodologies, spanning white-box gradients, multi-modal perturbations, and sophisticated multi-agent evolutionary strategies. Through an extensive empirical study on 20 advanced models (including GPT-5.2, Claude 4.5, and Gemini 3 Pro), we expose critical safety gaps: even frontier models fail to generalize across attack paradigms, with leading models exhibiting average Attack Success Rates as high as 49.14%. Notably, our findings reveal that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreaks. By open-sourcing OpenRT, we provide a sustainable, extensible, and continuously maintained infrastructure that accelerates the development and standardization of AI safety.
695e1350c03d6d81e439a05f
Xin Wang
claimed_verified
2026-01-07T09:25:09.928Z
677d32cf28895e2124065a63
/avatars/897fa9fb3b3adc6e60094e3b505f94bc.svg
Xin Wang
xinwang22
user
xinwang22
695e1350c03d6d81e439a060
Yunhao Chen
695e1350c03d6d81e439a061
Juncheng Li
695e1350c03d6d81e439a062
Yixu Wang
695e1350c03d6d81e439a063
Yang Yao
claimed_verified
2026-01-14T12:42:39.416Z
68e673f5f7b5b87f952b8568
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RvG3sSBXmWjVuGNwcvDGT.png
YANG YAO
EVIGBYEN
user
EVIGBYEN
695e1350c03d6d81e439a064
Tianle Gu
claimed_verified
2026-01-09T08:36:32.702Z
6346361c5efccdc07f179cae
/avatars/217818114a4c19ea4f3e5cdafefb625e.svg
Gu Tianle
Carol0110
user
Carol0110
695e1350c03d6d81e439a065
Jie Li
695e1350c03d6d81e439a066
Yan Teng
695e1350c03d6d81e439a067
Xingjun Ma
695e1350c03d6d81e439a068
Yingchun Wang
695e1350c03d6d81e439a069
Xia Hu
OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation. To address this, we introduce OpenRT, a unified, modular, and high-throughput red-teaming framework designed for comprehensive MLLM safety evaluation. At its core, OpenRT architects a paradigm shift in automated red-teaming by introducing an adversarial kernel that enables modular separation across five critical dimensions: model integration, dataset management, attack strategies, judging methods, and evaluation metrics. By standardizing attack interfaces, it decouples adversarial logic from a high-throughput asynchronous runtime, enabling systematic scaling across diverse models. Our framework integrates 37 diverse attack methodologies, spanning white-box gradients, multi-modal perturbations, and sophisticated multi-agent evolutionary strategies. Through an extensive empirical study on 20 advanced models (including GPT-5.2, Claude 4.5, and Gemini 3 Pro), we expose critical safety gaps: even frontier models fail to generalize across attack paradigms, with leading models exhibiting average Attack Success Rates as high as 49.14%. Notably, our findings reveal that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreaks. By open-sourcing OpenRT, we provide a sustainable, extensible, and continuously maintained infrastructure that accelerates the development and standardization of AI safety.
OpenRT is a unified red-teaming framework for evaluating multimodal large language model safety through modular adversarial testing across multiple attack dimensions and models.
Multimodal Large Language Models red-teaming adversarial kernel attack strategies judging methods evaluation metrics attack success rates multimodal perturbations multi-agent evolutionary strategies