Capacity and Trainability in Recurrent Neural Networks

TL;DR

The paper investigates two capacity bottlenecks in recurrent networks: how much task information can be stored in parameters and how much input history can be remembered by units. Through extensive hyperparameter optimization across six tasks and six RNN variants, it shows a roughly linear per-parameter capacity of about 5 bits and a per-unit input-memory capacity of about one real value, with little dependence on architecture. The authors find that training difficulty largely explains performance differences, with gated models being easier to train and sometimes more scalable to depth, while vanilla RNNs offer similar capacity but are harder to train. They introduce two new architectures, UGRNN and +RNN, that improve trainability in deep stacks and provide practical guidelines for architecture choice and compression based on training budgets and task complexity.

Abstract

Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.

Figures (6)

• Figure 1: All neural network architectures can store approximately five bits per parameter about a task, with only small variations across architectures. (a) Stored bits as a function of network size. These numbers represent the maximum stored bits across 1000+ HP optimizations with 5 time steps unrolled at each network size for all levels of depth. (b-d) Same as (a), but each level of depth shown separately. (e-h) Same as (a-d) but showing bits per parameter as a function of network size. (i) The value in cell $(x,y)$ is the multiplier for the number of parameters needed to give the architecture on the $x$-axis the same capacity as the architecture on the $y$-axis. Capacities are measured by averaging the maximum stored bits per parameter for each architecture across all sizes and levels of depth.
• Figure 2: Additional RNN capacity analysis. (a) The effect of the $\text{ReLU}$ nonlinearity on capacity. Solid lines indicate bits per parameter for 1-layer architectures (same as Figure \ref{['fig capacity tasks']}b), where both $\tanh$ and $\text{ReLU}$ are nonlinearity choices for the HP tuner. Dashed lines show the maximum bits per parameter for each architecture when only results achieved by the $\text{ReLU}$ nonlinearity are considered. (b) Bits per parameter as a function of the number of time steps unrolled. (c) L2 error curve for all architectures of all depths on the memory throughput task. The curve shows the error plotted as a function of the number of units for a random input of dimension 64 (black vertical line). All networks with with less than 64 units have error in reconstruction, while all networks with number of units greater than 64 nearly perfectly reconstruct the random input.
• Figure 3: All RNN architectures achieved near identical performance given the same number of parameters, on a language modeling and random function fitting task. (a-c) text8 Wikipedia number of parameters vs bits per character for all RNN architectures. From left to right: 1 layer, 2 layer, 4 layer models. (d) text8 number of hidden units vs bits per character for 1 layer architectures. We note that this is almost always a misleading way to compare architectures as the more heavily gated architectures appear to do better when compared per-unit. (e-g) Same as (a-c), except showing square error for different model sizes trained on RCFs.
• Figure 4: Some RNN architectures are far easier to train than others. Results of HP searches on extremely difficult tasks. (a) Median evaluation error as a function of HP optimization iteration for 1 layer architectures on the parentheses task. Dots indicate evaluation loss achieved on that HP iteration. (b-d) Same as (a), but for 2, 4 and 8 layer architectures. (e-h) Minimum evaluation error as a function of HP optimization iteration for parentheses task. Same depth order as (a-d). (i-p) Same as (a-h), except for the arithmetic task. We note that the best loss for the vanilla RNN is still decreasing after 2400+ HP evaluations.
• Figure 5: For randomly generated hyperparameters, GRU and +RNN are the most easily trainable architectures. Evaluation losses from 1000 iterations of randomly chosen HP sets for 1 and 8 layer, 200k parameter models on the parentheses task. Statistics from a Welch's $t$-test for equality of means on all pairs of architectures are presented in Table \ref{['table t test']}. (a) Box and whisker plot of evaluation losses for the 1 layer model. (b) Same as (a) but for 8 layers.