Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the locality bias and results in the Long Range Arena

Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, David Camacho

TL;DR

This paper scrutinizes the Long Range Arena benchmark and argues that many apparent gains from locality-biased architectures arise from task biases rather than genuine long-range modeling. With rotary positional encodings, data augmentation, and multitask denoising, Transformers reach near state-of-the-art on LRA, averaging $85.69\%$ across tasks, without a separate pretraining phase. In contrast, State Space Models (SSMs) and MEGA benefit mainly from inductive biases enforcing locality, while unrestricted long convolution kernels with proper data and training can match these gains. The authors call for redesigned benchmarks and standardized training protocols to disentangle data efficiency from true long-range dependency modeling.

Abstract

The Long Range Arena (LRA) benchmark was designed to evaluate the performance of Transformer improvements and alternatives in long-range dependency modeling tasks. The Transformer and its main variants performed poorly on this benchmark, and a new series of architectures such as State Space Models (SSMs) gained some traction, greatly outperforming Transformers in the LRA. Recent work has shown that with a denoising pre-training phase, Transformers can achieve competitive results in the LRA with these new architectures. In this work, we discuss and explain the superiority of architectures such as MEGA and SSMs in the Long Range Arena, as well as the recent improvement in the results of Transformers, pointing to the positional and local nature of the tasks. We show that while the LRA is a benchmark for long-range dependency modeling, in reality most of the performance comes from short-range dependencies. Using training techniques to mitigate data inefficiency, Transformers are able to reach state-of-the-art performance with proper positional encoding. In addition, with the same techniques, we were able to remove all restrictions from SSM convolutional kernels and learn fully parameterized convolutions without decreasing performance, suggesting that the design choices behind SSMs simply added inductive biases and learning efficiency for these particular tasks. Our insights indicate that LRA results should be interpreted with caution and call for a redesign of the benchmark.

On the locality bias and results in the Long Range Arena

TL;DR

This paper scrutinizes the Long Range Arena benchmark and argues that many apparent gains from locality-biased architectures arise from task biases rather than genuine long-range modeling. With rotary positional encodings, data augmentation, and multitask denoising, Transformers reach near state-of-the-art on LRA, averaging across tasks, without a separate pretraining phase. In contrast, State Space Models (SSMs) and MEGA benefit mainly from inductive biases enforcing locality, while unrestricted long convolution kernels with proper data and training can match these gains. The authors call for redesigned benchmarks and standardized training protocols to disentangle data efficiency from true long-range dependency modeling.

Abstract

The Long Range Arena (LRA) benchmark was designed to evaluate the performance of Transformer improvements and alternatives in long-range dependency modeling tasks. The Transformer and its main variants performed poorly on this benchmark, and a new series of architectures such as State Space Models (SSMs) gained some traction, greatly outperforming Transformers in the LRA. Recent work has shown that with a denoising pre-training phase, Transformers can achieve competitive results in the LRA with these new architectures. In this work, we discuss and explain the superiority of architectures such as MEGA and SSMs in the Long Range Arena, as well as the recent improvement in the results of Transformers, pointing to the positional and local nature of the tasks. We show that while the LRA is a benchmark for long-range dependency modeling, in reality most of the performance comes from short-range dependencies. Using training techniques to mitigate data inefficiency, Transformers are able to reach state-of-the-art performance with proper positional encoding. In addition, with the same techniques, we were able to remove all restrictions from SSM convolutional kernels and learn fully parameterized convolutions without decreasing performance, suggesting that the design choices behind SSMs simply added inductive biases and learning efficiency for these particular tasks. Our insights indicate that LRA results should be interpreted with caution and call for a redesign of the benchmark.
Paper Structure (24 sections, 6 equations, 3 figures, 7 tables)

This paper contains 24 sections, 6 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. The attention mechanism
  4. Long Range Arena
  5. New architectures
  6. State Space Models
  7. Synthesizing SSMs
  8. MEGA
  9. Improving the Transformer in the Long Range Arena
  10. gMLP
  11. Training method
  12. Transformer performance
  13. Removing restrictions from long-convolution kernels
  14. Discussion on the positional and local biases of the Long Range Arena tasks
  15. Ablation study: Transformer performance
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Distribution of labels based on the root operations in the ListOps task.
  • Figure 2: Accuracy difference between a Transformer model that achieves around 63% accuracy and a greedy algorithm that always predicts the most frequent label for each root operation (and randomly in the SUM MOD operation). The accuracies are reported by root operation.
  • Figure 3: Accuracy breakdown by operation and label for a Transformer model that achieves around 63% accuracy in the ListOps task.