Recurrent Neural Networks With Limited Numerical Precision

TL;DR

This work investigates training-time weight precision reductions for recurrent neural networks to enable efficient hardware implementations. By evaluating four quantization methods—BinaryConnect, TernaryConnect, Pow2-Ternarization, and Exponential Quantization—across vanilla RNNs, GRUs, and LSTMs on multiple datasets, it demonstrates that binarization often destabilizes RNNs while ternarization and fixed-point power-of-two quantization can preserve or even improve accuracy. The findings show notable gains on PTB and WSJ, and competitive results on TIDIGITS, highlighting a viable path to compute- and memory-efficient RNNs suitable for specialized hardware. This work provides practical, trainable low-precision strategies and releases code to facilitate adoption in hardware-oriented deployments.

Abstract

Recurrent Neural Networks (RNNs) produce state-of-art performance on many machine learning tasks but their demand on resources in terms of memory and computational power are often high. Therefore, there is a great interest in optimizing the computations performed with these models especially when considering development of specialized low-power hardware for deep networks. One way of reducing the computational needs is to limit the numerical precision of the network weights and biases. This has led to different proposed rounding methods which have been applied so far to only Convolutional Neural Networks and Fully-Connected Networks. This paper addresses the question of how to best reduce weight precision during training in the case of RNNs. We present results from the use of different stochastic and deterministic reduced precision training methods applied to three major RNN types which are then tested on several datasets. The results show that the weight binarization methods do not work with the RNNs. However, the stochastic and deterministic ternarization, and pow2-ternarization methods gave rise to low-precision RNNs that produce similar and even higher accuracy on certain datasets therefore providing a path towards training more efficient implementations of RNNs in specialized hardware.

Figures (3)

• Figure 1: Hidden state statistics when BinaryConnect, TernaryConnect, and exponential quantization are applied. (a) Maximum eigenvalue of Jacobian matrix of the hidden-to-hidden connection, and (b) Hidden state norms, at each time step. The vertical axis is shown in log scale. BinaryConnect yields much larger derivatives, which probably explain the failure to train the RNNs properly. This makes the activations blow up exponentially over time when using ReLU as the activation function. (b)
• Figure 2: Training and test set performance of vanilla RNNs learning on PTB (a) and text8 (b) datasets.
• Figure 3: Effect of ternarization methods on GRUs. Thick curves show the mean of 10 runs, half-transparent areas around a curve show the variance. Accuracy from ternarization on hidden-to-GRU weights (green), input-to-GRU weights (red), input-to-GRU weights and biases (magenta), and on all GRU weights and biases (black) compared against baseline (blue). (a) Pow2-ternarization. (b) Deterministic ternarization. Ternarization shows the same effects as pow2-ternarization, except for the case where all GRU weights and biases are ternarized (black).