Structured variational learning of Bayesian neural networks with horseshoe priors

Abstract

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection-even choosing the number of nodes-remains an open question. Recent work has proposed the use of a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. In this work, we propose several modeling and inference ad-vances that consistently improve the compactness of the model learned while maintaining' predictive performance, especially in smaller- sample settings including reinforcement learning.