SOTA Quality Estimation benchmarks and papers with code

Benchmarks

Dataset Name	SOTA Method	Metric
PAWS-X (test)	Sigmoid Head	BCE0.325	36	3mo ago
WMT en-de 22	COMET Kiwi	Pearson R0.722	15	3mo ago
WMT zh→en 2023 (test)		AUC81.3	12	8d ago
WMT 24	Sigmoid Head	Pearson Correlation0.606	12	3mo ago
Mandarin (zh-CN) dialect sentences (test)	NANO	Success Rate84	11	3mo ago
Portuguese (pt-BR) dialect sentences (test)	NANO	Success Rate86	11	3mo ago
EN-* Gender-ambiguous Fem. vs. Masc.	FairQE	QE Score (ES)99.47	10	1mo ago
CoNLL 2014 (test)	GRECO	Pearson Correlation (rho)0.445	10	3mo ago
FLORES-200 ordered 41,412 directions	ReMedy	Wins Count16.4	9	1d ago
En-Ml	ALOPE-RL	Pearson r0.583	9	3mo ago
WMT en-es 24	Sigmoid Head	Pearson Correlation0.672	8	3mo ago
WMT en-de 24	Sigmoid Head	Pearson Correlation0.606	8	3mo ago
ParaCrawl	COMET Kiwi	Pearson Correlation0.537	8	3mo ago
PAWS-X	Sigmoid Head	BCE0.883	8	3mo ago
EN-HI Gender-explicit	FairQE (ours, w/ MetricX 24 L)	Accuracy79.1	7	1mo ago
EN-DE Gender-explicit	FairQE (ours, w/ COMETKiwi 22)	Accuracy99.7	7	1mo ago
EN-AR Gender-explicit	FairQE (ours, w/ COMETKiwi 22)	Accuracy97.3	7	1mo ago
Gender-ambiguous EN-DE ES IT Neutral vs. Gendered	FairQE	QE Score (DE)0.9921	7	1mo ago
English-Indic QE Tourism domain		QE Score (En-Hi)0.737	6	2mo ago
Legal domain English-Indic QE		En-Ta QE Score0.749	6	2mo ago
Healthcare domain English-Indic QE		QE Score (En-Hi)0.611	6	2mo ago
English-Indic QE General domain		QE Score (En-Hi)0.563	6	2mo ago
WMT 2020 (test)	NANO	QE Score (en-cs)71.8	6	3mo ago
WMT RU-EN 2021 (test)	DAG 1	Pearson Correlation47.16	5	2mo ago
WMT RO-EN 2021 (test)	DAG 2	Pearson Correlation84.4	5	2mo ago

Showing 25 of 57 rows