Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models

TL;DR

The paper tackles the problem that merely increasing a model's context window does not guarantee improved long-context understanding due to many training samples lacking long-range semantic dependencies. It introduces ProLong, a data mining framework that assigns Long Dependency Scores to documents by integrating dependency strength, distance, and specificity, then uses efficient sampling to scale to large corpora. Empirical results show ProLong-selected data improves long-context tasks across language modeling, retrieval, and real-world benchmarks, often outperforming full-data training with only half the data. This approach demonstrates that high-quality long-dependency data can significantly boost long-context capabilities, suggesting a shift toward data-centric strategies for extending LLM context windows.

Abstract

Long-context modeling capabilities are important for large language models (LLMs) in various applications. However, directly training LLMs with long context windows is insufficient to enhance this capability since some training samples do not exhibit strong semantic dependencies across long contexts. In this study, we propose a data mining framework \textbf{ProLong} that can assign each training sample with a long dependency score, which can be used to rank and filter samples that are more advantageous for enhancing long-context modeling abilities in LLM training. Specifically, we first use delta perplexity scores to measure the \textit{Dependency Strength} between text segments in a given document. Then we refine this metric based on the \textit{Dependency Distance} of these segments to incorporate spatial relationships across long-contexts. Final results are calibrated with a \textit{Dependency Specificity} metric to prevent trivial dependencies introduced by repetitive patterns. Moreover, a random sampling approach is proposed to optimize the computational efficiency of ProLong. Comprehensive experiments on multiple benchmarks indicate that ProLong effectively identifies documents that carry long dependencies and LLMs trained on these documents exhibit significantly enhanced long-context modeling capabilities.

Figures (5)

• Figure 1: Samples that carry longer dependencies better enhances LLMs' long-context modeling capabilities, even with a fixed training context window of 32k.
• Figure 2: ProLong first segments a training sample $S$ into $N$ equal-length portions, then computing three key metrics: (a) dependency strength, (b) dependency distance, and (c) dependency specificity. These metrics are integrated via Eq.\ref{['eq:lds']} to derive the long dependency score for $S$.
• Figure 3: Visualization results of the dependency strength heat map for four data instances of equal length.
• Figure 4: 0k-32k pressure test (Needle In A HayStack) performance of ProLong-7B and ProLong-13B.
• Figure 5: Repetitive patterns examples from arXiv, C++, PG19 datasets.