Short-Context Dominance: How Much Local Context Natural Language Actually Needs?

TL;DR

This work empirically tests the short-context dominance hypothesis in natural language by introducing Minimal Context Length (MCL), Distributionally Aware MCL (DaMCL), and long-context detection (LSDS) to quantify when local prefixes suffice for next-token prediction. It finds that 75-80% of sequences with 1k–7k tokens rely on the last 32–96 tokens, a pattern robust across languages and domains. To operate at inference time without ground-truth, DaMCL and LSDS provide practical detection of long-context dependence, and a targeted boosting method TaBoo leverages long-context signals to improve QA performance across models and tasks. Overall, the paper contributes a principled framework and actionable decoding techniques that mitigate short-context bias and enable more context-aware language generation.

Abstract

We investigate the short-context dominance hypothesis: that for most sequences, a small local prefix suffices to predict their next tokens. Using large language models as statistical oracles, we measure the minimum context length (MCL) needed to reproduce accurate full-context predictions across datasets with sequences of varying lengths. For sequences with 1-7k tokens from long-context documents, we consistently find that 75-80% require only the last 96 tokens at most. Given the dominance of short-context tokens, we then ask whether it is possible to detect challenging long-context sequences for which a short local prefix does not suffice for prediction. We introduce a practical proxy to MCL, called Distributionally Aware MCL (DaMCL), that does not require knowledge of the actual next-token and is compatible with sampling strategies beyond greedy decoding. Our experiments validate that simple thresholding of the metric defining DaMCL achieves high performance in detecting long vs. short context sequences. Finally, to counter the bias that short-context dominance induces in LLM output distributions, we develop an intuitive decoding algorithm that leverages our detector to identify and boost tokens that are long-range-relevant. Across Q&A tasks and model architectures, we confirm that mitigating the bias improves performance.

Figures (35)

• Figure 1: Short-context dominance hypothesis. In this example, with context 1 the model fails to predict the ground-truth "neighbour," Context 2 produces a semantically valid alternative "friend," and when using Context 3, the model correctly predicts "neighbour." Our work (1) systematically validates the hypothesis, (2) develops methods to detect when longer context is truly needed, and (3) leverages these insights to improve language model sampling by correcting short-context bias.
• Figure 2: Distribution of MCL: Minimum context window needed to confidently predict the next token across sampled sequences, six datasets, and three LLMs. $\hat{b}$ denotes the slope of the log-log fit. Blue/green distinguishes between long ($\geq$ 6k tokens) and short ($\geq$ 1k tokens) documents, respectively.
• Figure 3: Distribution of DaMCL: DaMCL measurements for various sampling strategies and relative thresholds. Results are shown separately for short- (top row) and long- (bottom row) documents to highlight potential differences in behavior. While the overall trend resembles the heavy-tailed decaying pattern observed in standard MCL (see Fig. \ref{['fig:sec1_MCL_Hist']}), the choice of threshold influences the outcome. Each subplot reflects results aggregated over all model–dataset combinations, as only minor deviations were observed across different configurations under identical hyperparameters.
• Figure 4: DaMCL distribution for nucleus sampling with fixed sub-context increments. Still heavily biased towards short context.
• Figure 5: Distribution of $\mathsf{LSDS}\left(\mathbf{s}\right)$ values for short- and long-context sequences for Mistral-7B-Instruct on controlled validation setup.

Theorems & Definitions (4)

• Definition 1
• Definition 2
• Definition 3
• Definition 4