Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Colored MNIST

Benchmarks

Task NameDataset NameSOTA ResultTrend
Image ClassificationColored-MNIST (CMNIST)
Accuracy94.43
42
Image classificationColored MNIST unbiased (test)
Accuracy98.02
28
Digit ClassificationColored MNIST foreground color (test)
Unbiased Accuracy86.37
24
Image ClassificationColored MNIST (test)
Accuracy75
24
Counterfactual Distribution EstimationColored MNIST a=4
Mean Out-Sample W220.44
16
Counterfactual Distribution EstimationColored MNIST a=3
Mean Out-Sample W218.39
16
Counterfactual Distribution EstimationColored MNIST a=2
Mean Out-Sample W218.23
16
Counterfactual Distribution EstimationColored MNIST a=1
Mean Out-Sample W217.62
16
Counterfactual Distribution EstimationColored MNIST a=0
Mean Out-Sample W220.29
16
Image ClassificationColored MNIST (unbiased)
Accuracy0.9628
16
Image ClassificationColored MNIST (bias-guiding)
Accuracy100
16
Image ClassificationColored MNIST (train)
Accuracy88.9
14
Image ClassificationColored MNIST
Accuracy (+90% Threshold)73.1
9
Image ClassificationColored MNIST background color, ratio 0.95
Bias-Aligned Accuracy99.9
5
Image ClassificationColored MNIST background color, ratio 0.98
Bias-Aligned Accuracy99.97
5
Image ClassificationColored MNIST background color, ratio 0.99
Bias Aligned Score100
5
Image ClassificationColored MNIST background color, ratio 0.995
Aligned Accuracy1
5
Digit ClassificationColored MNIST (val)
Validation Accuracy (Kendall's tau)1
5
Color ClassificationColored-MNIST (val)
Unbiased Accuracy99.95
4
OOD DetectionColored MNIST In-distribution vs Textures protocol (test)
TNR @ 95% TPR100
2
OOD DetectionColored MNIST In-distribution vs iSUN protocol (test)
TNR @ 95% TPR100
2
OOD DetectionColored MNIST In-distribution vs LSUN protocol (test)
TNR @ 95% TPR1
2
OOD DetectionColored MNIST In-distribution vs Spurious OOD protocol (test)
TNR @ 95% TPR0.958
2
Showing 23 of 23 rows