Preference Evaluation

Benchmarks

Dataset Name	SOTA Method	Metric
Mturk user study Overall Preferences 1.0	Policy R1	Overall Preference Fraction76	30	3d ago
SpeechJudge	UrgentMOS	Acc@0.575	15	3d ago
SpeechEval	UrgentMOS	Acc@0.583	15	3d ago
URGENT SQA 24	UrgentMOS	Acc@0.559	15	3d ago
TMHINT-QI	UrgentMOS	Acc@0.563	15	3d ago
SOMOS	UrgentMOS	Acc@0.560	15	3d ago
URGENT25-SQA	UrgentMOS	Acc@0.559	15	3d ago
CHiME UDASE 7 (test)	UrgentMOS	Acc@0.566	15	3d ago
NISQA-FOR	UrgentMOS	Acc@0.581	15	3d ago
NISQA-P501	SCOREQ	Acc@0.581	15	3d ago
ImageReward	BLPO	F1 Score34	15	3d ago
Detail	Pluralistic	Avg Score8.25	14	3d ago
OCR-like	Ours (Homo.)	Average Score9.06	14	3d ago
Medical	Ours (Homo.)	Avg Score8.58	14	3d ago
AlpacaEval 2	SelectiveDPO	WR (%)559	12	3d ago
PPE Preference (test)	Probe	Kuiper Statistic0.0434	8	3d ago
HH-Helpful	DLMA-13B	Win Count52	8	3d ago
HH-Harmless	DLMA-13B	Win Rate60	8	3d ago
PKU-SafeRLHF	DLMA-13B	Win Rate57	8	3d ago
Anthropic-SafeRLHF (target)	πbias (rubric-based preference attack)	Win Rate41.7	2	3d ago
Anthropic-SafeRLHF benchmark	πbias (rubric-based preference attack)	Win Rate33.7	2	3d ago
Ultra-Real (target)	πbias (rubric-based preference attack)	Win Rate43	2	3d ago
Ultra-Real benchmark	πbias (rubric-based preference attack)	Win Rate43.1	2	3d ago

Showing 23 of 23 rows