Deep Variational Information Bottleneck
About
We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, Kevin Murphy• 2016
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Named Entity Recognition | CoNLL 2003 (test) | -- | 539 | |
| Image Classification | MNIST (test) | Test Accuracy97.6 | 189 | |
| Image Classification | CINIC-10 (test) | -- | 177 | |
| Out-of-Distribution Detection | CIFAR-10 vs SVHN (test) | AUROC0.97 | 101 | |
| Out-of-Distribution Detection | CIFAR-10 vs CIFAR-100 (test) | AUROC88 | 93 | |
| Image Classification | CIFAR-10N (Worst) | Accuracy78.88 | 83 | |
| Image Classification | CIFAR-10N (Aggregate) | Accuracy86.11 | 78 | |
| Out-of-Distribution Detection | CIFAR-10 in-distribution LSUN out-of-distribution (test) | AUROC96 | 73 | |
| Named Entity Recognition | WNUT 2017 (test) | F1 Score51.6 | 63 | |
| Image Classification | CIFAR-10-C (test) | -- | 61 |
Showing 10 of 52 rows