Share your thoughts, 1 month free Claude Pro on usSee more

Safety Overrefusal Evaluation on Overrefusal Evaluation Suite (test)

1XSTest

Gemma2-9B

Updated 3mo ago

Evaluation Results

Method	Links
Gemma2-9B 2026.02		1	0.974	0.987
Augmented RT 2026.02		1	0.974	0.987
Llama3-8B 2026.02		0.98	0.989	0.985
Gemma2-2B 2026.02		0.98	0.975	0.978
Fail-Closed Alignment 2026.02		0.92	0.983	0.952
Fail-Closed Alignment 2026.02		0.88	0.971	0.926
LAT 2026.02		0.76	0.944	0.852
Fail-Closed Alignment 2026.02		0.76	0.962	0.861
Llama2-7B 2026.02		0.75	0.993	0.872
LAT 2026.02		0.74	0.937	0.839
CAT 2026.02		0.72	0.986	0.853
DeepRefusal 2026.02		0.7	0.965	0.833
CAT 2026.02		0.64	0.969	0.805
DeepRefusal 2026.02		0.62	0.965	0.793
DeepRefusal 2026.02		0.6	0.958	0.779
LAT 2026.02		0.6	0.978	0.789
Fail-Closed Alignment 2026.02		0.46	0.974	0.717
Augmented RT 2026.02		0.4	0.974	0.687
DeepRefusal 2026.02		0.34	0.901	0.621
Augmented RT 2026.02		0.3	0.966	0.633
Augmented RT 2026.02		0.26	0.968	0.614
CAT 2026.02		0.2	0.959	0.58
CAT 2026.02		0.18	0.85	0.515
LAT 2026.02		0.1	0.868	0.484