Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

NATS-Bench

Benchmarks

Task NameDataset NameSOTA ResultTrend
Architecture performance predictionNATS-Bench SSS ImageNet16-120 (stratified subset)
Mean Abs Kendall Corr Coeff85.6
24
Architecture performance predictionNATS-Bench SSS CIFAR-100 (stratified subset)
mAKCC3.4
24
Architecture performance predictionNATS-Bench SSS CIFAR-10 (stratified subset)
mAKCC88.8
24
Image ClassificationNATS-Bench SSS ImageNet-16-120 (test)
Accuracy46.2
20
Image ClassificationNATS-Bench SSS CIFAR-100 (val)
Accuracy73.49
20
Image ClassificationNATS-Bench SSS ImageNet-16-120 (val)
Top-1 Accuracy46.73
13
Image ClassificationNATS-Bench SSS CIFAR-100 (test)
Top-1 Accuracy73.51
13
Image ClassificationNATS-Bench SSS CIFAR-10 (test)
Top-1 Acc94.37
13
Image ClassificationNATS-Bench SSS CIFAR-10 (val)
Top-1 Accuracy91.61
13
NAS Proxy EvaluationNATS-Bench TSS ImageNet16-120 Stratified Subset
Mean Abs Spearman Correlation84.7
12
NAS Proxy EvaluationNATS-Bench TSS CIFAR-100 stratified subset
Spearman Correlation (Abs)0.885
12
NAS Proxy EvaluationNATS-Bench TSS (Topology Search Space) CIFAR-10 stratified
Mean Abs Spearman Correlation92.8
12
NAS Proxy EvaluationNATS-Bench SSS (Size Search Space) ImageNet16-120 stratified subset
Mean Absolute Spearman Correlation Coefficient0.971
12
NAS Proxy EvaluationNATS-Bench SSS (Size Search Space) CIFAR-100 stratified subset
mAbs Spearman Correlation0.924
12
NAS Proxy EvaluationNATS-Bench SSS (Size Search Space) CIFAR-10 stratified subset
Spearman Correlation (x100)98.2
12
Zero-cost Proxy EvaluationNATS-Bench TSS ImageNet16-120 (non-stratified)
Mean Absolute Kendall Correlation54.7
12
Zero-cost Proxy EvaluationNATS-Bench TSS CIFAR-100 (non-stratified)
mAbs Kendall Correlation0.609
12
Zero-cost Proxy EvaluationNATS-Bench TSS CIFAR-10 (non-stratified)
Mean Absolute Kendall Correlation0.611
12
Zero-cost Proxy EvaluationNATS-Bench SSS ImageNet16-120 (non-stratified)
mKendall Corr (abs)0.795
12
Zero-cost Proxy EvaluationNATS-Bench SSS CIFAR-100 (non-stratified)
mAbs Kendall Corr4.4
12
Zero-cost Proxy EvaluationNATS-Bench SSS CIFAR-10 (non-stratified)
mAbs Kendall Corr (x100)0.768
12
Neural Architecture Search (Topology Search)NATS-Bench ImageNet16-120 (test)
Accuracy46.58
10
Neural Architecture Search (Topology Search)NATS-Bench CIFAR-100 (test)
Accuracy73.01
10
Neural Architecture Search (Topology Search)NATS-Bench CIFAR-10 (test)
Accuracy94.34
10
Neural Architecture Search (Size Search)NATS-Bench ImageNet16-120 (test)
Accuracy45.37
4
Showing 25 of 40 rows