Learning Longer-term Dependencies in RNNs with Auxiliary Losses

TL;DR

The paper tackles the challenge of learning long-range dependencies in RNNs by introducing unsupervised auxiliary losses that either reconstruct past subsequences or predict future subsequences around randomly chosen anchors. This framework enables effective training with truncated BPTT, improves optimization and regularization, and delivers strong performance on very long sequences (up to 16,000 steps) across image-like and text benchmarks, while remaining memory- and computation-efficient. Although Transformer models can outperform on some short-sequence tasks, the proposed r-LSTM/p-LSTM methods demonstrate robust scalability and competitive accuracy as sequence length grows, with notable advantages in memory usage and training time. The work suggests a practical, orthogonal augmentation to recurrent models that can be combined with other scalable architectures to handle ultra-long sequences in real-world settings.

Abstract

Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences. This paper proposes a simple method that improves the ability to capture long term dependencies in RNNs by adding an unsupervised auxiliary loss to the original objective. This auxiliary loss forces RNNs to either reconstruct previous events or predict next events in a sequence, making truncated backpropagation feasible for long sequences and also improving full BPTT. We evaluate our method on a variety of settings, including pixel-by-pixel image classification with sequence lengths up to 16\,000, and a real document classification benchmark. Our results highlight good performance and resource efficiency of this approach over competitive baselines, including other recurrent models and a comparable sized Transformer. Further analyses reveal beneficial effects of the auxiliary loss on optimization and regularization, as well as extreme cases where there is little to no backpropagation.

Figures (6)

• Figure 1: An overview of our method. The auxiliary loss improves the memory of the recurrent network such that the number of steps needed for the main task's BPTT is small.
• Figure 2: An overview of our methods. For each random anchor point, say F, we build an auxiliary loss at its position. Left: We predict a random subsequence BCD that occurs before F. B is inserted into a decoder network to start the reconstruction, while C and D is optionally fed. Right: We predict the subsequence GHI by stacking an auxiliary RNNs on top of the main one. Gradients from auxiliary loss is truncated in both cases to keep the overall cost of BPTT constant.
• Figure 3: Top: Test accuracy on StanfordDogs resized to 8 levels of sequence length. Bottom: Time to run a single mini batch of 128 training examples, measured in second.
• Figure 4: Effects of shrinking supervised BPTT length.
• Figure 5: Effects of auxiliary losses on training/testing accuracy.