R-Transformer: Recurrent Neural Network Enhanced Transformer

TL;DR

R-Transformer addresses the dual challenge of modeling local structure and long-range dependencies in sequences by pairing a LocalRNN that encodes local neighborhoods with a multi-head attention module that captures global context, all without position embeddings. The architecture stacks $N$ layers of LocalRNN, LayerNorm, MultiHeadAttention, and FeedForward blocks, enabling parallelization and robust performance. Across MNIST, Nottingham, and Penn Treebank, R-Transformer consistently outperforms Transformer, TCN, and recurrent baselines, demonstrating strong gains in both locality signaling and global dependencies. This work suggests that explicitly combining locality-aware processing with global attention yields substantial practical benefits for diverse sequence modeling tasks, and it releases code to facilitate adoption and further research.

Abstract

Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. We have made the code publicly available at \url{https://github.com/DSE-MSU/R-transformer}.

Figures (2)

• Figure 1: The illustration of one layer of R-Transformer. There are three different networks that are arranged hierarchically. In particular, the lower-level is localRNNs that process positions in a local window sequentially (This figure shows an example of local window of size 3); The middle-level is multi-head attention networks which capture the global long-term dependencies; The upper-level is Position-wise feedforward networks that conduct non-linear feature transformation. These three networks are connected by a residual and layer normalization operation. The circles with dash line are the paddings of the input sequence
• Figure 2: An illustration of the original and local RNN. In contrast to orignal RNN which maintains a hidden state at each position summarizing all the information seen so far, LocalRNN only operates on positions within a local window. At each position, LocalRNN will produce a hidden state that represents the information in the local window ending at that position.