LLM$\times$MapReduce: Simplified Long-Sequence Processing using Large Language Models

TL;DR

This work proposes a novel training-free framework for processing long texts, utilizing a divide-and-conquer strategy to achieve comprehensive document understanding, and designs a structured information protocol to better cope with inter-chunk dependency and an in-context confidence calibration mechanism to resolve inter-chunk conflicts.

Abstract

Enlarging the context window of large language models (LLMs) has become a crucial research area, particularly for applications involving extremely long texts. In this work, we propose a novel training-free framework for processing long texts, utilizing a divide-and-conquer strategy to achieve comprehensive document understanding. The proposed LLM$\times$MapReduce framework splits the entire document into several chunks for LLMs to read and then aggregates the intermediate answers to produce the final output. The main challenge for divide-and-conquer long text processing frameworks lies in the risk of losing essential long-range information when splitting the document, which can lead the model to produce incomplete or incorrect answers based on the segmented texts. Disrupted long-range information can be classified into two categories: inter-chunk dependency and inter-chunk conflict. We design a structured information protocol to better cope with inter-chunk dependency and an in-context confidence calibration mechanism to resolve inter-chunk conflicts. Experimental results demonstrate that LLM$\times$MapReduce can outperform representative open-source and commercial long-context LLMs, and is applicable to several different models.

Figures (4)

• Figure 1: Overview of the proposed LLM$\times$MapReduce framework. After dividing the provided long text into a series of chunks, the model processes each chunk to extract an information structure containing the essential content needed to address the query. This is referred to as the map stage in our framework. The mapped results are then compressed during the collapse stage, preparing them for the reduce stage. The structure of the collapsed results mirrors that of the mapped results. The collapse stage ensures that the input to the reducing model remains within its effective length (i.e., $L$). Based on the structured outputs from the first two stages (i.e., the map and collapse stages), the reduce model aggregates information from all chunks, resolves inter-chunk conflicts using calibrated confidence scores, and predicts the final answer.
• Figure 2: Prompt for in-context confidence calibration.
• Figure 3: Performance of Llama3-70B-Instruct$\times$MapReduce on the NIAH test, with the maximum length of the haystack set to 1280K tokens.
• Figure 4: Comparison of inference latency. "L3-70B-G" represents Llama3-70B-Instruct-Gradient-1048K.