Make Your LLM Fully Utilize the Context

TL;DR

This paper tackles the lost-in-the-middle problem in long-context LLMs by proposing Information-Intensive Training (In2), a data-driven approach that synthesizes long-context QA data to teach models that crucial information can reside anywhere in a long context. Applying In2 to Mistral-7B yields FILM-7B, which demonstrates robust long-context information retrieval across document, code, and structured-data contexts via VaL Probing, and achieves competitive performance with GPT-4-Turbo on probing tasks and strong gains on real-world long-context benchmarks like NarrativeQA. Importantly, FILM-7B maintains short-context performance, indicating the approach enhances long-context usage without sacrificing existing capabilities. The work also introduces a comprehensive evaluation suite (VaL Probing) that more accurately assesses long-context utilization than traditional probes, contributing a scalable data-generation pipeline and insights into training strategies like RoPE scaling and the limits of sliding-window methods.

Abstract

While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middle challenge. We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information. Based on this intuition, our study presents information-intensive (IN2) training, a purely data-driven solution to overcome lost-in-the-middle. Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the integration and reasoning of information from two or more short segments. Through applying this information-intensive training on Mistral-7B, we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window. Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2 accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.

Figures (6)

• Figure 2: The data construction process for In2 training, aimed at enhancing the fine-grained information awareness (upper), and the integration and reasoning of information (lower).
• Figure 3: Three tasks in VaL Probing. The retrieval patterns are determined by the relative positions between the HTML]A9D6FBretrieval keywords and the information to be retrieved.
• Figure 4: Performance of FilM-7B on VaL Probing and the comparisons with (a) Mistral, (b) LongAlign, and (c) InternLM2. The X-axis is the relative position in the context ($\sim$32K tokens).
• Figure 5: Performances of FilM-7B and the backbone model on short-context tasks.
• Figure 6: Performance of FilM-7B with a 4K sliding window (SW). PT-In2: apply the sliding window in both pre-training and In2 training. In2: apply the sliding window only in In2 training.