XSTest

Benchmarks

Task Name	Dataset Name	SOTA Result
Over-refusal	XSTest	Overrefusal Rate0	102
Safety Evaluation	XSTest Unsafe	False Refusal Rate (FR)0	84
Safety Evaluation	XSTest Safe	FC4	78
Response Harmfulness Detection	XSTEST-RESP	Response Harmfulness F195.48	76
Over-refusal Evaluation	XSTest	Evaluation Score (avg@4)100	70
Refusal Rate Evaluation	XSTest Safe (test)	Refusal Rate0	56
Safety Classification	XSTest	F1 Score95.8	46
Over-refusal Evaluation	XSTest Safe	Over-refusal Rate1.6	45
Safety Evaluation	XSTest	F1 Score97	44
Safety Evaluation	XSTest (test)	XSTest Score95	36
Safety Evaluation	XSTest	FRR0.4	35
Adversarial and Jailbreaking Attack Detection	XSTest	AUROC0.984	35
Safety Evaluation	XSTest (combined)	F1 Score100	34
Safety Evaluation	XSTest	Safety Score98.4	32
Safety Evaluation	XSTest	Accuracy96	30
Refusal Evaluation	XSTest Unsafe	Refusal Rate100	25
Safety Alignment	XSTest	Compliance95.2	21
Harmful Content Safety Evaluation	XsTest Harmful	Attack Success Rate (ASR)0	20
Harmful prompt detection	XSTest	F1 Score97.44	20
Harmlessness	XsTest	Refusal Rate99	20
Safety Classification	XSTest (test)	F192.91	20
Safety Evaluation	XsTest	Harmful Rate2	16
Response Classification	XSTest Text Response	F1 Score98.43	16
Prompt classification	XSTest	F1 Score94.8	16
Safety Evaluation	XSTest Toxic	Safety95	15

Showing 25 of 84 rows