SOTA Preference Evaluation benchmarks and papers with code

Benchmarks

Dataset Name	SOTA Method	Metric
AlpacaEval 2	SelectiveDPO	WR (%)559	64	1mo ago
VCD	SFT	Multi-choice One Preference p+94.15	39	3mo ago
Mturk user study Overall Preferences 1.0	Policy R1	Overall Preference Fraction76	30	4mo ago
ImageReward		Accuracy67.5	29	1mo ago
RMB Best-of-N	Skywork-Reward-V2-Llama-3.1-8B-40M	Helpfulness Score (BoN)86.2	16	4mo ago
SpeechJudge	UrgentMOS	Acc@0.575	15	4mo ago
SpeechEval	UrgentMOS	Acc@0.583	15	4mo ago
URGENT SQA 24	UrgentMOS	Acc@0.559	15	4mo ago
TMHINT-QI	UrgentMOS	Acc@0.563	15	4mo ago
SOMOS	UrgentMOS	Acc@0.560	15	4mo ago
URGENT25-SQA	UrgentMOS	Acc@0.559	15	4mo ago
CHiME UDASE 7 (test)	UrgentMOS	Acc@0.566	15	4mo ago
NISQA-FOR	UrgentMOS	Acc@0.581	15	4mo ago
NISQA-P501	SCOREQ	Acc@0.581	15	4mo ago
Detail	Pluralistic	Avg Score8.25	14	4mo ago
OCR-like	Ours (Homo.)	Average Score9.06	14	4mo ago
Medical	Ours (Homo.)	Avg Score8.58	14	4mo ago
UltraFeedback core250 (test)	TEA	Win Rate80	12	2mo ago
PPE Preference (test)	Probe	Kuiper Statistic0.0434	8	4mo ago
HH-Helpful	DLMA-13B	Win Count52	8	4mo ago
HH-Harmless	DLMA-13B	Win Rate60	8	4mo ago
PKU-SafeRLHF	DLMA-13B	Win Rate57	8	4mo ago
Anthropic-SafeRLHF (target)	πbias (rubric-based preference attack)	Win Rate41.7	2	4mo ago
Anthropic-SafeRLHF benchmark	πbias (rubric-based preference attack)	Win Rate33.7	2	4mo ago
Ultra-Real (target)	πbias (rubric-based preference attack)	Win Rate43	2	4mo ago

Showing 25 of 27 rows