SOTA Preference Evaluation benchmarks and papers with code

Benchmarks

Dataset Name	SOTA Method	Metric
VCD	SFT	Multi-choice One Preference p+94.15	39	4d ago
Mturk user study Overall Preferences 1.0	Policy R1	Overall Preference Fraction76	30	1mo ago
ImageReward	Q-Real-Score	Accuracy60.7	20	17d ago
RMB Best-of-N	Skywork-Reward-V2-Llama-3.1-8B-40M	Helpfulness Score (BoN)86.2	16	1mo ago
SpeechJudge	UrgentMOS	Acc@0.575	15	1mo ago
SpeechEval	UrgentMOS	Acc@0.583	15	1mo ago
URGENT SQA 24	UrgentMOS	Acc@0.559	15	1mo ago
TMHINT-QI	UrgentMOS	Acc@0.563	15	1mo ago
SOMOS	UrgentMOS	Acc@0.560	15	1mo ago
URGENT25-SQA	UrgentMOS	Acc@0.559	15	1mo ago
CHiME UDASE 7 (test)	UrgentMOS	Acc@0.566	15	1mo ago
NISQA-FOR	UrgentMOS	Acc@0.581	15	1mo ago
NISQA-P501	SCOREQ	Acc@0.581	15	1mo ago
Detail	Pluralistic	Avg Score8.25	14	1mo ago
OCR-like	Ours (Homo.)	Average Score9.06	14	1mo ago
Medical	Ours (Homo.)	Avg Score8.58	14	1mo ago
AlpacaEval 2	SelectiveDPO	WR (%)559	14	3d ago
PPE Preference (test)	Probe	Kuiper Statistic0.0434	8	1mo ago
HH-Helpful	DLMA-13B	Win Count52	8	1mo ago
HH-Harmless	DLMA-13B	Win Rate60	8	1mo ago
PKU-SafeRLHF	DLMA-13B	Win Rate57	8	1mo ago
Anthropic-SafeRLHF (target)	πbias (rubric-based preference attack)	Win Rate41.7	2	1mo ago
Anthropic-SafeRLHF benchmark	πbias (rubric-based preference attack)	Win Rate33.7	2	1mo ago
Ultra-Real (target)	πbias (rubric-based preference attack)	Win Rate43	2	1mo ago
Ultra-Real benchmark	πbias (rubric-based preference attack)	Win Rate43.1	2	1mo ago

Showing 25 of 25 rows