The NLP Task Effectiveness of Long-Range Transformers

TL;DR

Transformers impose $O(N^2)$ time and space complexity for long sequences, motivating long-range variants that use sparsified patterns, recurrence memories, or kernel-based attention. The paper conducts a fair, large-scale evaluation of seven long-range Transformer variants across five NLP tasks and seven datasets, aiming to isolate the impact of long-range attention from pretraining and hyperparameters. Through attribution, entropy, and breakdown analyses, it reveals that long-range attention can improve content selection and query-guided decoding in some tasks, but may underutilize distant tokens and suffer from cumulative kernel approximation errors in deeper networks. The findings offer practical guidance: use pattern-based models with global tokens for guidance-based tasks, rely on recurrence memories for general improvements, and exercise caution with kernel-based methods due to error accumulation; the work also notes significant limitations, including energy costs and language bias, guiding future work toward more robust, scalable long-range transformers.

Abstract

Transformer models cannot easily scale to long sequences due to their O(N^2) time and space complexity. This has led to Transformer variants seeking to lower computational complexity, such as Longformer and Performer. While such models have theoretically greater efficiency, their effectiveness on real NLP tasks has not been well studied. We benchmark 7 variants of Transformer models on 5 difficult NLP tasks and 7 datasets. We design experiments to isolate the effect of pretraining and hyperparameter settings, to focus on their capacity for long-range attention. Moreover, we present various methods to investigate attention behaviors to illuminate model details beyond metric scores. We find that the modified attention in long-range transformers has advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error.

Figures (11)

• Figure 1: Illustration of 5 patterns used by long-range transformers, from beltagy2021ParagraphsNLPLong with permission.
• Figure 2: Recurrent transformers. "No Grad" means that the gradients do not back-propagate to this block. Obtained from dai2019TransformerXLAttentiveLanguage with permission.
• Figure 3: B$^3$ breakdown scores of 4 models for mention pairs with ranges from $[1, 8)$ to $[512, \infty)$.
• Figure 4: The breakdown analysis of DocNLI. We pick the best configuration for each model for brevity.
• Figure 5: The performance of Longformer and BigBird on different lengths of TriviaQA documents. Note that Longformer$^G$ and BigBird ($L$=$\infty$) have global tokens.