Malicious Prompt Refusal on HarmBench

96Refusal Rate

microsoft/Phi-3-mini-128k-instruct

Updated 1mo ago

Evaluation Results

Method
microsoft/Phi-3-mini-128k-instruct 2025.12	96	-
Phi-3-mini-128k-Instruct 2026.06	96	-
google/gemma-2b-it 2025.12	94	-
Gemma-2B-IT 2026.06	94	-
meta-llama/Llama-3.1-8B-instruct 2025.12	89	-
Llama-3.1-8B-Instruct 2026.06	89	-
meta-llama/Llama-3.2-3B-instruct 2025.12	84	-
Llama-3.2-3B-Instruct 2026.06	84	-
SafeAnchor 2026.04	78.4	-
lmsys/vicuna-7b-v1.5 2025.12	76	0.32
Vicuna-7B-v1.5 2026.06	76	32
google/gemma-2b-it 2025.12	62	0.91
Gemma-2B-IT 2026.06	62	91
microsoft/Phi-3-mini-128k-instruct 2025.12	59	0.86
Phi-3-mini-128k-Instruct 2026.06	59	86
lmsys/vicuna-7b-v1.5 2025.12	58	0.86
Vicuna-7B-v1.5 2026.06	58	86
meta-llama/Llama-3.1-8B-instruct 2025.12	55	0.88
meta-llama/Llama-3.2-3B-instruct 2025.12	55	0.9
Llama-3.1-8B-Instruct 2026.06	55	88
Llama-3.2-3B-Instruct 2026.06	55	90
SafeGrad + LoRA 2026.04	54.6	-
meta-llama/Llama-3.2-3B-instruct 2025.12	52	0.2
Llama-3.2-3B-Instruct 2026.06	52	20
google/gemma-2b-it 2025.12	51	0.35
lmsys/vicuna-7b-v1.5 2025.12	51	-
microsoft/Phi-3-mini-128k-instruct 2025.12	51	0.17
Gemma-2B-IT 2026.06	51	35
Vicuna-7B-v1.5 2026.06	51	-
Phi-3-mini-128k-Instruct 2026.06	51	17
Safety Interleaving 2026.04	49.3	-
Vaccine + LoRA 2026.04	46.5	-
Safe LoRA 2026.04	44.7	-
EWC + LoRA 2026.04	36.4	-
O-LoRA 2026.04	34.8	-
Standard LoRA 2026.04	31.2	-
meta-llama/Llama-3.1-8B-instruct 2025.12	20	0.23
Llama-3.1-8B-Instruct 2026.06	20	23