Are Long-LLMs A Necessity For Long-Context Tasks?

TL;DR

The paper argues that most long-context tasks can be solved using short-context reasoning rather than expanding model context, formalizing this via a long-context problem where $|\mathcal{X}| \gg L$. It introduces LC-Boost, a bootstrapping framework that enables a short-context LLM to access and utilize relevant parts of a long input through a discrete action space, enabling both retrieval-based and sequential processing. Theoretical analysis using the data-processing inequality and empirical evidence show that decomposing long contexts into short chunks can approximate the minimal sufficient context $\tilde{\mathcal{X}}$ and yield performance comparable to or better than brute-force long-context methods, with significant energy and token savings. Across 12 datasets, LC-Boost demonstrates strong performance while reducing resource consumption, highlighting a path toward greener, scalable long-context reasoning.

Abstract

The learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we argue that the long-LLMs are not a necessity to solve long-context tasks, as common long-context tasks are short-context solvable, i.e. they can be solved by purely working with oracle short-contexts within the long-context tasks' inputs. On top of this argument, we propose a framework called LC-Boost (Long-Context Bootstrapper), which enables a short-LLM to address the long-context tasks in a bootstrapping manner. In our framework, the short-LLM prompts itself to reason for two critical decisions: 1) how to access to the appropriate part of context within the input, 2) how to make effective use of the accessed context. By adaptively accessing and utilizing the context based on the presented tasks, LC-Boost can serve as a general framework to handle diversified long-context processing problems. We comprehensively evaluate different types of tasks from popular long-context benchmarks, where LC-Boost is able to achieve a substantially improved performance with a much smaller consumption of resource.

Figures (4)

• Figure 1: Illustration for LC-Boost. The LLM is prompted to reason for how to access to proper context and how to utilize the accessed context to solve the task. Toy Examples. (A) Brute-force solution. Despite correctness, it is unnecessarily expensive due to the processing of the entire context simultaneously. (B) Naive RAG. It is hard to handle problems like information aggregation, which leads to the incomplete answer. (C) LC-Boost leverages RAG to tackle the problem, which produces the correct answer in a small cost. (D) LC-Boost processes the long-context via sequential scan, which correctly solves the problem based on the comprehensively collected information.
• Figure 2: Pilot Study Across Various Tasks: In the Brute-force setting, the entire context is processed by GPT-4-128K. In the LC-Boost setting, the maximum context length is restricted to 4K, and LC-Boost is utilized to solve the long-context problem with short context.
• Figure 3: Performance comparison on different context processing strategies in the ablation study. NarrativeQA (left) is a single-doc QA task. HotpotQA (middle) is a multi-doc QA task. SamSUM (right) is a few-shot learning task.
• Figure 4: Energy consumption analysis.