Share your thoughts, 1 month free Claude Pro on usSee more

Agent Safety on ASSEBench

92.04Accuracy

DRAFT (Qwen3Guard-Gen-4B)

Updated 1d ago

Evaluation Results

Method
DRAFT (Qwen3Guard-Gen-4B) 2026.02	92.04	90.55	-	88.77
DRAFT (Qwen3-8B) 2026.02	91.57	89.75	-	87.19
Gemini-2 2026.05	91.5	91.44	-	-
DRAFT (Qwen3-4B-Instruct-2507) 2026.02	91.38	88.98	-	86.04
Extractor 2026.02	90.81	88.09	96.06	81.33
DRAFT 2026.02	89.97	87.23	93.18	82
DRAFT (Llama-3.1-8B) 2026.02	89.72	86.91	-	84.62
DRAFT 2026.02	89.69	86.25	97.48	77.33
QwQ-32B 2026.05	89.63	90.09	-	-
GPT-4.1 2026.05	89.12	88.37	-	-
Claude-3.5 2026.05	89.02	89.44	-	-
Extractor 2026.02	88.86	86.01	90.44	82
Deepseek v3 2026.05	88.66	87.81	-	-
GPT-o3-mini 2026.05	87.99	86.95	-	-
DRAFT 2026.02	87.19	83.45	90.62	77.33
GPT-4o 2026.05	85.63	84.73	-	-
Qwen-2.5-32B 2026.05	85.19	85.7	-	-
DRAFT 2026.02	84.4	80.69	83.57	78.01
Qwen3-4B-Instruct-2507 (SFT) 2026.02	84.22	77.06	-	68.9
SFT 2026.02	84.12	77.99	92.66	67.33
Extractor 2026.02	83.01	77.15	88.03	68.67
ShieldAgent 2026.05	82.33	82.92	-	-
Qwen-2.5-7B 2026.05	81.53	80.53	-	-
Qwen3Guard-Gen-4B (SFT) 2026.02	81.09	74.15	-	68.34
Extractor 2026.02	81.06	71.43	96.59	56.67
Qwen3-8B (AgentAuditor) 2026.02	80.82	81.84	-	79.33
Qwen3-8B (SFT) 2026.02	80.17	72.27	-	64.45
GPT-4.1 2026.05	79.69	78.17	-	-
SFT 2026.02	79.39	70.16	88.78	58
GPT-o3-mini 2026.05	79.37	76.63	-	-
Claude-3.5 2026.05	79.31	81.08	-	-
Extractor 2026.02	78.83	69.6	87.06	58.03
Llama-3.1-8B (AgentAuditor) 2026.02	77.99	79.41	-	70.67
Deepseek v3 2026.05	77.58	74.6	-	-
DRAFT 2026.02	77.16	67.2	84.04	56.02
Llama-3.1-8B (SFT) 2026.02	76.56	63.41	-	57.17
QwQ-32B 2026.05	76.3	78.44	-	-
ChatGPT 5.2 (API) 2026.02	74.67	71.6	-	62.56
Qwen3Guard-Gen-4B (AgentAuditor) 2026.02	74.19	72.44	-	67.01
SFT 2026.02	73.54	59.23	83.13	46
Gemini-2 2026.05	72.74	65.6	-	-
GPT-4o 2026.05	72.19	69	-	-
SFT 2026.02	71.59	52.34	87.5	37.33
SFT 2026.02	71.31	52.09	86.15	37.33
Qwen3-4B-Instruct-2507 (AgentAuditor) 2026.02	71.06	58.92	-	60.3
Llama-3.1-8B 2026.05	70.81	74.9	-	-
gpt-oss-120b (Vanilla) 2026.02	69.52	67.85	-	60.51
Llama-Guard-3 2026.05	68.54	74.62	-	-
Qwen-2.5-32B 2026.05	65.51	68.37	-	-
Llama-3.1-8B (LoRA) 2026.02	65.18	39.02	-	26.67
Qwen3-8B (LoRA) 2026.02	64.76	57.91	-	57.33
Qwen3-4B-Instruct-2507 (LoRA) 2026.02	63.79	65.24	-	81.33
Qwen3-4B-Instruct-2507 (Vanilla) 2026.02	63.23	44.57	-	41.09
Qwen3Guard-Gen-4B (Vanilla) 2026.02	62.64	34.36	-	23.66
Llama-3.1-8B (Vanilla) 2026.02	61.55	25.69	-	16.08
LoRA 2026.02	60.45	39.83	54.65	31.33
LoRA 2026.02	59.61	18.08	59.26	10.67
Qwen3Guard-Gen-4B (LoRA) 2026.02	59.33	29.81	-	20.67
Vanilla 2026.02	58.96	35.98	50.64	27.91
Vanilla 2026.02	58.8	14.41	50.92	8.39
Qwen3-8B (Vanilla) 2026.02	58.69	49.87	-	49.85
Qwen-2.5-7B 2026.05	57.41	56.16	-	-
LoRA 2026.02	56.82	59.95	48.95	77.33
Vanilla 2026.02	54.2	57.49	46.63	74.92
Vanilla 2026.02	54.12	55.94	46.37	70.48
LoRA 2026.02	51.25	56.36	45.02	75.33
Llama-3.1-8B 2026.05	51.02	65.19	-	-
Vanilla 2026.02	50.56	50.44	43.06	60.87
LoRA 2026.02	50.14	50.42	43.13	60.67