SOTA Over-refusal evaluation benchmarks and papers with code

Benchmarks

Dataset Name	SOTA Method	Metric
NQ (Natural Questions)	VectorSteer	ORR0	72	4mo ago
ORFuzzSet	LLM-VA	ORR16	72	4mo ago
XSTest		Evaluation Score (avg@4)100	70	22d ago
XSTest Safe	AdaCD	Over-refusal Rate1.6	45	1mo ago
Over-refusal (test)	No train	Refusal Rate0	36	3mo ago
MMMU in-scope (test)	Prompt-based	Math Score37	32	4mo ago
ScienceQA in-scope (test)	System Prompt	Biology Refusal Count0	32	4mo ago
Poisoned GPT4-LLM subset		Informative Refusal Rate35.7	27	25d ago
OR-Bench (boundary cases)	Mistral-v2-7B	OR-FPR1.7	18	2mo ago
XSTest, CoCoNot	HARC	XSTest Refusal Rate2.6	16	23d ago
Over-Refusal Evaluation OrBench-Hard-1k FalseReject MorBench OverBench	YuFeng-XGuard-0.6B	ORR (OrBench-Hard-1k)23.22	11	1mo ago
OverBench	w/o Safety-Aligned Optimization	ORR5.59	11	1mo ago
MorBench	FreoStream	ORR1.47	11	1mo ago
FalseReject	FreoStream	ORR9.12	11	1mo ago
XS (test)	THRD-style	OR Score22	7	22d ago
WildGuard Unharmful	Categorical Steering	Over-refusal Rate1.06	7	4mo ago
CoCoNot Contrast	Categorical Steering	Over-refusal Rate1.58	7	4mo ago
XsTest Safe (test)		Over-Refusal Rate3.5	5	1mo ago
OR-Bench-Hard-1K (test)	Robust Harmful Features detector	F1 Score60.6	4	26d ago
Benign prompt dataset	Base	Over-Refusal Rate17	4	3mo ago
XSTest (test)	Claude Sonnet 4.5	Over-refusal Rate0.035	4	4mo ago

Showing 21 of 21 rows