Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simple Local Attentions Remain Competitive for Long-Context Tasks

Wenhan Xiong, Barlas Oğuz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, Yashar Mehdad

TL;DR

The paper addresses the challenge of long-context processing in Transformer-based NLP by analyzing multiple efficient attention variants under a standard pretraining-finetuning setup, highlighting the quadratic attention cost $O(L^2)$ and the need for scalable solutions. Using a unified RoBERTa-like encoder and large-scale pretraining on long documents, the authors compare fixed local, learnable sparse, kernel-based/low-rank, and hybrid attentions, with and without global tokens. They find that, with proper pretraining, simple local window attention often matches or outperforms more complex long-range mechanisms, and that disjoint local blocks can achieve comparable downstream performance with about half the pretraining compute. The work advocates for careful, practical evaluation of long-context models and demonstrates that local attentions can be both efficient and effective, providing code to reproduce the experiments.

Abstract

Many NLP tasks require processing long contexts beyond the length limit of pretrained models. In order to scale these models to longer text sequences, many efficient long-range attention variants have been proposed. Despite the abundance of research along this direction, it is still difficult to gauge the relative effectiveness of these models in practical use cases, e.g., if we apply these models following the pretrain-and-finetune paradigm. In this work, we aim to conduct a thorough analysis of these emerging models with large-scale and controlled experiments. For each attention variant, we pretrain large-size models using the same long-doc corpus and then finetune these models for real-world long-context tasks. Our findings reveal pitfalls of an existing widely-used long-range benchmark and show none of the tested efficient attentions can beat a simple local window attention under standard pretraining paradigms. Further analysis on local attention variants suggests that even the commonly used attention-window overlap is not necessary to achieve good downstream results -- using disjoint local attentions, we are able to build a simpler and more efficient long-doc QA model that matches the performance of Longformer~\citep{longformer} with half of its pretraining compute. The code to replicate our experiments can be found at https://github.com/pytorch/fairseq/tree/main/examples/xformers

Simple Local Attentions Remain Competitive for Long-Context Tasks

TL;DR

The paper addresses the challenge of long-context processing in Transformer-based NLP by analyzing multiple efficient attention variants under a standard pretraining-finetuning setup, highlighting the quadratic attention cost and the need for scalable solutions. Using a unified RoBERTa-like encoder and large-scale pretraining on long documents, the authors compare fixed local, learnable sparse, kernel-based/low-rank, and hybrid attentions, with and without global tokens. They find that, with proper pretraining, simple local window attention often matches or outperforms more complex long-range mechanisms, and that disjoint local blocks can achieve comparable downstream performance with about half the pretraining compute. The work advocates for careful, practical evaluation of long-context models and demonstrates that local attentions can be both efficient and effective, providing code to reproduce the experiments.

Abstract

Many NLP tasks require processing long contexts beyond the length limit of pretrained models. In order to scale these models to longer text sequences, many efficient long-range attention variants have been proposed. Despite the abundance of research along this direction, it is still difficult to gauge the relative effectiveness of these models in practical use cases, e.g., if we apply these models following the pretrain-and-finetune paradigm. In this work, we aim to conduct a thorough analysis of these emerging models with large-scale and controlled experiments. For each attention variant, we pretrain large-size models using the same long-doc corpus and then finetune these models for real-world long-context tasks. Our findings reveal pitfalls of an existing widely-used long-range benchmark and show none of the tested efficient attentions can beat a simple local window attention under standard pretraining paradigms. Further analysis on local attention variants suggests that even the commonly used attention-window overlap is not necessary to achieve good downstream results -- using disjoint local attentions, we are able to build a simpler and more efficient long-doc QA model that matches the performance of Longformer~\citep{longformer} with half of its pretraining compute. The code to replicate our experiments can be found at https://github.com/pytorch/fairseq/tree/main/examples/xformers
Paper Structure (28 sections, 2 figures, 9 tables)

This paper contains 28 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminaries of Tested Attention Variants
  3. Fixed local patterns.
  4. Learnable sparse attention patterns.
  5. Kernel-based/Low-rank methods.
  6. Hybrid attention.
  7. A note on global tokens.
  8. Experiment Setup
  9. LRA Experiments
  10. Pretraining and Downstream Tasks
  11. Pretraining Resource.
  12. Pretraining Architecture
  13. Downstream Datasets and Metrics.
  14. Task-specific Architectures for Finetuning.
  15. Results and Analysis
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Attention pattern visualization of two types of local attentions: Left: Local window attention as in Longformer, with window size 2; Right: Blockwise local window attention with block size 2. The rows represent the tokens in the sequence and the columns represent the tokens being attended to.
  • Figure 2: Pretraining curves of the non-overlapping block attentions with various context windows.