What Context Features Can Transformer Language Models Use?

TL;DR

This study investigates why transformers benefit from long-range contexts by using the V-information framework to measure usable information and by conducting targeted ablations on WikiText-103 with a GPT-2–style model. It shows that long-range predictive power largely rests on content words and local co-occurrence statistics rather than detailed syntax or topic signals, since aggressive manipulations of word order or function-word content often remove little usable information. The results suggest that more efficient context representations, rather than simply longer contexts, could improve language modeling, and they highlight nuances between training/evaluation paradigms and the role of overfitting. The findings have practical implications for designing scalable, information-preserving context mechanisms in future transformer-based LMs.

Abstract

Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of experiments that measure usable information by selectively ablating lexical and structural information in transformer language models trained on English Wikipedia. In both mid- and long-range contexts, we find that several extremely destructive context manipulations -- including shuffling word order within sentences and deleting all words other than nouns -- remove less than 15% of the usable information. Our results suggest that long contexts, but not their detailed syntactic and propositional content, are important for the low perplexity of current transformer language models.

Figures (7)

• Figure 1: Calculation of the ablated likelihood $\mathcal{L}(\texttt{nouns}, \ell: m\sim n)$ (\ref{['eq:semi-batched']}). A context ablation nouns (which deletes all non-noun words) is applied to the first $\ell$ tokens of the context, and likelihood is computed on the last $n-m$ (unablated) context tokens.
• Figure 2: Effect of word order on usable information. Bar labels show "change in ablated likelihood (ablated information)". The $x$ axis shows ablated likelihood. Error bars represent 95% confidence intervals. Word-order changes that preserve local ordering remove only a small amount of information, while shuffling or replacement with thematically similar text remove more.
• Figure 3: Effect of word identity on usable information. Labels are as in \ref{['fig:order']}. Several ablations, including deletion of all words except nouns, preserve most usable information in the mid-range condition, and improve model accuracy in the in the long range.
• Figure 4: Loss of information resulting from ablations at evaluation time only. $x$-axis and labels show ablated negative log-likelihoods. Some locality-preserving ablations (high PMI, shuf. sent.) have a small effect, but most affect likelihood significantly (including lexical ablations that do not remove usable information).
• Figure 5: Comparison of model performance in the train+eval and eval-only settings. The units represent the percentage of the gap between the full information and no information models/contexts. That way, if a point falls on the dotted $y=x$ line, then that ablation has the same relative effect in each paradigm. If a point falls above the dotted line, then that ablation leads to better relative performance in the train+eval paradigm, and if a point falls below the dotted line, then that ablation leads to better relative performance in the eval-only paradigm.

Theorems & Definitions (2)

• Definition 1
• Definition 2