Bayesian Recurrent Neural Networks

TL;DR

The paper proposes a scalable Bayesian treatment for recurrent neural networks by applying Bayes by Backprop (BBB) with truncated backpropagation through time, enabling uncertainty estimates and regularisation with reduced parameter count. A novel posterior sharpening technique introduces a batch-conditioned refinement of the weight posterior, reducing gradient variance and improving learning. The authors demonstrate improved perplexities in language modeling and stronger captioning metrics in image captioning compared with non-Bayesian and baseline Bayesian approaches, and they provide a new uncertainty benchmark. The work also positions its methods as broadly applicable to neural networks beyond RNNs, offering insights into uncertainty calibration and model averaging in deep architectures.

Abstract

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

Figures (6)

• Figure 1: Illustration (left) and Algorithm (right) of Bayes by Backprop applied to an RNN.
• Figure 2: Weight pruning experiment. No significant loss on performance is observed until pruning more than 80% of weights.
• Figure 3: Entropy gap $\Delta H_p$ (Eq. \ref{['eq:entropy_gap']}) between reversed and regular Penn Treebank test sets $\times$ number of samples.
• Figure 4: Image captioning results on MSCOCO development set.
• Figure 5: Pruning patterns for one LSTM cell (with 650 untis) from converged model with 80% of total weights dropped. A white dot indicates that particular parameter was dropped. In the middle column, a horizontal white line means that row was set to zero. Finally, the last column indicates the total number of weights removed for each row.