Performance Prediction

Benchmarks

Dataset Name	SOTA Method	Metric
WMT Benchmarks Average WMT'14 & WMT'19 (aggregation)	LLM-PP	MAE0.29	16	1mo ago
WMT En-De 2019 (val)	LLM-PP	MAE0.29	16	1mo ago
WMT En-Fr 2014 (val)	LLM-PP	MAE0.28	16	1mo ago
WMT En-De 2014 (val)	LLM-Distill-PP	MAE0.22	16	1mo ago
ARC 1.2k (test)	Metabench	MAE1.14	11	1mo ago
Winogrande (WG) 1.3k (test)	DISCO	MAE1	11	1mo ago
HellaSwag (HS) 10k (test)	Metabench	MAE0.8	11	1mo ago
MMLU 14k (test)	DISCO	MAE1.07	11	1mo ago
Large Model Performance Prediction Dataset 80% masking (test)	STAR	RMSE7.5	10	1mo ago
MNLI source domains (out-of-domain)	Cosine distance (fine-tuned)	ROC AUC0.683	10	1mo ago
MNLI source domains (in-domain)	Cosine distance (fine-tuned)	ROC AUC0.699	10	1mo ago
Sentiment temporal (out-of-domain)	Cosine distance (fine-tuned)	ROC AUC0.834	10	1mo ago
Sentiment temporal (in-domain)	Cosine distance (fine-tuned)	ROC AUC0.852	10	1mo ago
Sentiment categories (out-of-domain)	Cosine distance (fine-tuned)	ROC AUC0.822	10	1mo ago
Sentiment categories (in-domain)	Cosine distance (fine-tuned)	ROC AUC0.845	10	1mo ago
Tatoeba		MAE5.82	9	1mo ago
MewsliX	MAML	MAE9.33	9	1mo ago
LAREQA		MAE1.51	9	1mo ago
XQUAD	MDGPR	MAE3.15	9	1mo ago
TyDiQA		MAE4.29	9	1mo ago
XCOPA	MDGPR	MAE1.96	9	1mo ago
PAWS		MAE0.8	9	1mo ago
MLQA	Group Lasso	MAE2.21	9	1mo ago
Performance Prediction Evaluation Suite 70B Model on GSM8k, MATH, BBH, TriviaQA, MBPP, AGIEval, DROP, MMLU-pro (evaluation sets)	COD (Complete)	Mean Absolute Prediction Error (%)1.55	6	1mo ago
ARC-LoRA (test)	W2T	MAE0.32	5	1mo ago

Showing 25 of 33 rows