SOTA Calibration benchmarks and papers with code

Benchmarks

Dataset Name	SOTA Method	Metric
CIFAR-100 (test)	DPS	ECE0.85	104	2mo ago
HMDA (test)		Oracle ECE1.29	100	21d ago
BEAR (test)	gemma-7b	Brier Score0.083	96	3mo ago
MMLU	Verbalized confidence	Brier Score0.0559	58	14d ago
USPS		ECE1.54	57	12d ago
NQ	Temp. Scaling	ECE0.046	55	3mo ago
CIFAR-10H	P+L (Recalibrated)	ECE0.84	52	3mo ago
Brock-Hommes (test)	ANTR	MSE0	40	3mo ago
TriviaQA	Probe (train on TriviaQA)	Brier Score0.0845	39	2d ago
MNIST	ETS	ECE0.21	33	3mo ago
TruthfulQA		Gain2.132	32	2d ago
SQuAD	Temp. Scaling	ECE5.87	31	3mo ago
WebQ	Temp. Scaling	ECE0.0674	31	3mo ago
Digital-S	Knowledge-Transferring-based Temperature Scaling	ECE7.37	27	3mo ago
Average StrategyQA, HotpotQA, NQ, Bamboogle	NAACL	ECE0.264	24	3mo ago
Bamboogle	NAACL	ECE0.113	24	3mo ago
HotpotQA	NAACL	ECE0.28	24	3mo ago
StrategyQA	NAACL	ECE0.285	24	3mo ago
ImageNet-C OOD-domains	MD-TS	ECE (%)1.43	24	3mo ago
ImageNet-C (InD-domains)	TS	ECE (%)0.89	24	3mo ago
CIFAR-10 5000-sample half (test)		ECE0.0095	23	22d ago
Tabular datasets	Platt	NLL0.2983	21	8d ago
stream 2,000-question	Verbalized + Temp Scaling	ECE0.08	21	1mo ago
CelebA ImageResNet (test)		ECE (Oracle Estimate)0.39	20	21d ago
CIFAR100 Noise levels 1-5 (val)	Deep Ensemble	NLL1.499	20	3mo ago

Showing 25 of 100 rows