A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5
The rapid evolution of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has produced substantial gains in reasoning, perception, and generative capability across language and vision. However, whether these advances yield commensurate improvements in safety remains unclear, in part due to fragmented evaluation practices limited to single modalities or threat models. In this report, we present an integrated safety evaluation of 7 frontier models: GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5. We evaluate each model across language, vision-language, and image generation settings using a unified protocol that integrates benchmark evaluation, adversarial evaluation, multilingual evaluation, and compliance evaluation. Aggregating our evaluations into safety leaderboards and model safety profiles across multiple evaluation modes reveals a sharply heterogeneous safety landscape. While GPT-5.2 demonstrates consistently strong and balanced safety performance across evaluations, other models exhibit pronounced trade-offs among benchmark safety, adversarial alignment, multilingual generalization, and regulatory compliance. Both language and vision-language modalities show significant vulnerability under adversarial evaluation, with all models degrading substantially despite strong results on standard benchmarks. Text-to-image models achieve relatively stronger alignment in regulated visual risk categories, yet remain brittle under adversarial or semantically ambiguous prompts. Overall, these results show that safety in frontier models is inherently multidimensional--shaped by modality, language, and evaluation scheme, underscoring the need for standardized safety evaluations to accurately assess real-world risk and guide responsible model development and deployment.
6969a8ce32f0333869ff93c9
Xingjun Ma
6969a8ce32f0333869ff93ca
Yixu Wang
6969a8ce32f0333869ff93cb
Hengyuan Xu
claimed_verified
2026-01-16T10:29:47.929Z
634bde123d11eaedd889e277
https://cdn-avatars.huggingface.co/v1/production/uploads/1665916392312-noauth.png
Hengyuan Xu
DobyXu
user
DobyXu
6969a8ce32f0333869ff93cc
Yutao Wu
6969a8ce32f0333869ff93cd
Yifan Ding
6969a8ce32f0333869ff93ce
Yunhan Zhao
6969a8ce32f0333869ff93cf
Zilong Wang
6969a8ce32f0333869ff93d0
Jiabin Hua
claimed_verified
2026-03-25T10:52:08.056Z
655101623fe6c0b1f8b58987
/avatars/4d36a4988e6011fec3ceac2b59938c3a.svg
Jiabin Hua
Ammmob
user
Ammmob
6969a8ce32f0333869ff93d1
Ming Wen
6969a8ce32f0333869ff93d2
Jianan Liu
6969a8ce32f0333869ff93d3
Ranjie Duan
6969a8ce32f0333869ff93d4
Yifeng Gao
6969a8ce32f0333869ff93d5
Yingshui Tan
6969a8ce32f0333869ff93d6
Yunhao Chen
6969a8ce32f0333869ff93d7
Hui Xue
6969a8ce32f0333869ff93d8
Xin Wang
claimed_verified
2026-01-16T10:29:50.416Z
677d32cf28895e2124065a63
/avatars/897fa9fb3b3adc6e60094e3b505f94bc.svg
Xin Wang
xinwang22
user
xinwang22
6969a8ce32f0333869ff93d9
Wei Cheng
6969a8ce32f0333869ff93da
Jingjing Chen
6969a8ce32f0333869ff93db
Zuxuan Wu
6969a8ce32f0333869ff93dc
Bo Li
6969a8ce32f0333869ff93dd
Yu-Gang Jiang
A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5
The rapid evolution of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has produced substantial gains in reasoning, perception, and generative capability across language and vision. However, whether these advances yield commensurate improvements in safety remains unclear, in part due to fragmented evaluation practices limited to single modalities or threat models. In this report, we present an integrated safety evaluation of 7 frontier models: GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5. We evaluate each model across language, vision-language, and image generation settings using a unified protocol that integrates benchmark evaluation, adversarial evaluation, multilingual evaluation, and compliance evaluation. Aggregating our evaluations into safety leaderboards and model safety profiles across multiple evaluation modes reveals a sharply heterogeneous safety landscape. While GPT-5.2 demonstrates consistently strong and balanced safety performance across evaluations, other models exhibit pronounced trade-offs among benchmark safety, adversarial alignment, multilingual generalization, and regulatory compliance. Both language and vision-language modalities show significant vulnerability under adversarial evaluation, with all models degrading substantially despite strong results on standard benchmarks. Text-to-image models achieve relatively stronger alignment in regulated visual risk categories, yet remain brittle under adversarial or semantically ambiguous prompts. Overall, these results show that safety in frontier models is inherently multidimensional--shaped by modality, language, and evaluation scheme, underscoring the need for standardized safety evaluations to accurately assess real-world risk and guide responsible model development and deployment.
Frontier language and vision models show varied safety performance across different evaluation criteria, highlighting the need for comprehensive, standardized safety assessments.
Large Language Models Multimodal Large Language Models safety evaluation adversarial evaluation multilingual evaluation compliance evaluation safety leaderboards model safety profiles