NExtLong: Toward Effective Long-Context Training without Long Documents

TL;DR

NExtLong addresses the scarcity of long documents for training ultra-long context LLMs by introducing Negative Document Extension, which interleaves hard negative chunks between meta-chunks to strengthen long-range dependency modeling. The method relies on chunking documents into meta-chunks, mining hard negatives from a deduplicated pretraining corpus via FAISS, and training with a next-token objective that emphasizes distinguishing relevant long-range context from distractors. Empirical results on HELMET and RULER show consistent, significant gains over prior long-context data-synthesis methods and competitive performance against SOTA models, including strong ICL behavior without relying on authentic long documents. The approach reduces dependence on long documents and demonstrates promise for training ultra-long context LLMs, with future work aimed at diversifying negative chunks and extending domain coverage.

Abstract

Large language models (LLMs) with extended context windows have made significant strides yet remain a challenge due to the scarcity of long documents. Existing methods tend to synthesize long-context data but lack a clear mechanism to reinforce the long-range dependency modeling. To address this limitation, we propose NExtLong, a novel framework for synthesizing long-context data through Negative document Extension. NExtLong decomposes a document into multiple meta-chunks and extends the context by interleaving hard negative distractors retrieved from pretraining corpora. This approach compels the model to discriminate long-range dependent context from distracting content, enhancing its ability to model long-range dependencies. Extensive experiments demonstrate that NExtLong achieves significant performance improvements on the HELMET and RULER benchmarks compared to existing long-context synthesis approaches and leading models, which are trained on non-synthetic long documents. These findings highlight NExtLong's ability to reduce reliance on non-synthetic long documents, making it an effective framework for developing advanced long-context LLMs.

Figures (9)

• Figure 1: Comparison of existing remarkable models and NExtLong on the HELMETyen2024helmet and RULERhsieh2024ruler benchmarks. We evaluate various task types classified by HELMET. All results are averaged over sequence lengths of 8K, 16K, 32K, 64K, and 128K.
• Figure 2: The NExtLong method consists of two stages. In the first stage, a document is chunked into multiple meta-chunks, and each meta-chunk is mined for numerous hard negatives. These hard negatives are then concatenated with the meta-chunks to create a long document. In the second stage, the model is trained using this synthesized long document, focusing on modeling long-range dependencies by identifying the meta-chunks across a wide range of hard negatives.
• Figure 3: Comparison of NExtLong with other data synthesis methods on HELMET and RULER benchmarks across different context lengths. NExtLong shows significant performance improvements across various tasks.
• Figure 4: Comparison of NExtLong with GPT-4o on five In-Context Learning (ICL) tasks from the HELMET benchmark. Each polyline represents the model's performance across context lengths of 8K, 16K, 32K, 64K, and 128K.
• Figure 5: NExtLong enhances long-range dependency modeling. The bars represent the model's ability to capture long-range dependencies, measured by the attention weights assigned to the first third of the context. The dotted line indicates the model’s performance, demonstrating a positive correlation between improved long-range dependency modeling and better performance on the LongQA task.