MM-SafetyBench
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Safety Evaluation | MM-SafetyBench | Average ASR0 | 98 | |
| Jailbreak Attack Defense | MM-SafetyBench | Attack Success Rate (ASR)0.2 | 56 | |
| Video Jailbreaking | MM-SafetyBench 1.0 (test) | Attack Success Rate96 | 48 | |
| Jailbreaking Attack | MM-SafetyBench | Attack Success Rate (ASR)95 | 42 | |
| Multimodal Jailbreak Attack | MM-SafetyBench (full) | ASR95.42 | 40 | |
| Jailbreaking MLLMs | MM-SafetyBench | WASR100 | 32 | |
| Jailbreak Attack | MM-SafetyBench (tiny) | ASR99.16 | 25 | |
| Safety Evaluation | MM-SafetyBench v1.0 (test) | ASR0.6 | 24 | |
| Multimodal Safety Evaluation | MM-SafetyBench | Safety Score2.73 | 22 | |
| Safety Evaluation | MM-SafetyBench (test) | Helpfulness Score68.95 | 20 | |
| Jailbreak Attack Success Evaluation | MM-SafetyBench SD+TYPO | ASR81.4 | 18 | |
| Jailbreak Attack Success Evaluation | MM-SafetyBench TYPO | Attack Success Rate (ASR)79.6 | 18 | |
| Jailbreak Attack Success Evaluation | MM-SafetyBench SD | ASR80.6 | 18 | |
| Direct Malicious | MM-SafetyBench OOD | ASR0.71 | 16 | |
| Response Safety | MM-SafetyBench (avg) | MS-R99 | 15 | |
| MLLM Jailbreaking | MM-SafetyBench Physical Harm scenario | ASR6 | 15 | |
| Multimodal Jailbreak Defense | MM-SafetyBench (full) | ASR (Illegal Activity - S)1.03 | 12 | |
| Multimodal Safety Defense | MM-SafetyBench SD_TYPO | Average ASR12 | 10 | |
| Multimodal Safety Defense | MM-SafetyBench SD | Average ASR0.09 | 10 | |
| Harmful Rate Evaluation | MM-SafetyBench OCR (test) | Illegal Activity Rate0 | 10 | |
| Safety Evaluation | MM-SafetyBench (MMSB) | Attack Success Rate (V-T AVG)1.8 | 9 | |
| Structured-based Jailbreak Attack Defense | MM-SafetyBench unseen attack types | ASR (SD)2.35 | 9 | |
| Multimodal Large Language Model Safety Evaluation | MM-SafetyBench++ | Illegal Activity Unsafe Refusal Rate100 | 9 | |
| Jailbreak Detection | MM-SafetyBench | AUROC99.18 | 9 | |
| Multimodal Safety Evaluation | MM-SafetyBench SD + TYPO + SD_TYPO (test) | ASR Score0.08 | 8 |
Showing 25 of 35 rows