SOTA Preference Evaluation benchmarks and papers with code

Benchmarks

Dataset Name	SOTA Method	Metric
AlpacaEval 2	SelectiveDPO	WR (%)559	48	2d ago
VCD	SFT	Multi-choice One Preference p+94.15	39	1mo ago
Mturk user study Overall Preferences 1.0	Policy R1	Overall Preference Fraction76	30	3mo ago
ImageReward	Q-Real-Score	Accuracy60.7	20	2mo ago
RMB Best-of-N	Skywork-Reward-V2-Llama-3.1-8B-40M	Helpfulness Score (BoN)86.2	16	3mo ago
SpeechJudge	UrgentMOS	Acc@0.575	15	3mo ago
SpeechEval	UrgentMOS	Acc@0.583	15	3mo ago
URGENT SQA 24	UrgentMOS	Acc@0.559	15	3mo ago
TMHINT-QI	UrgentMOS	Acc@0.563	15	3mo ago
SOMOS	UrgentMOS	Acc@0.560	15	3mo ago
URGENT25-SQA	UrgentMOS	Acc@0.559	15	3mo ago
CHiME UDASE 7 (test)	UrgentMOS	Acc@0.566	15	3mo ago
NISQA-FOR	UrgentMOS	Acc@0.581	15	3mo ago
NISQA-P501	SCOREQ	Acc@0.581	15	3mo ago
Detail	Pluralistic	Avg Score8.25	14	3mo ago
OCR-like	Ours (Homo.)	Average Score9.06	14	3mo ago
Medical	Ours (Homo.)	Avg Score8.58	14	3mo ago
UltraFeedback core250 (test)	TEA	Win Rate80	12	21d ago
PPE Preference (test)	Probe	Kuiper Statistic0.0434	8	3mo ago
HH-Helpful	DLMA-13B	Win Count52	8	3mo ago
HH-Harmless	DLMA-13B	Win Rate60	8	3mo ago
PKU-SafeRLHF	DLMA-13B	Win Rate57	8	3mo ago
Anthropic-SafeRLHF (target)	πbias (rubric-based preference attack)	Win Rate41.7	2	3mo ago
Anthropic-SafeRLHF benchmark	πbias (rubric-based preference attack)	Win Rate33.7	2	3mo ago
Ultra-Real (target)	πbias (rubric-based preference attack)	Win Rate43	2	3mo ago

Showing 25 of 27 rows