Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, Xiaojiang Liu

TL;DR

This work investigates how the spacing and distribution of multiple relevant information pieces in long-context inputs bias LLMs. It introduces LongPiBench, a benchmark that isolates absolute and relative positional effects across four input lengths up to 256K tokens and evaluates nine models. The findings show reduced susceptibility to the traditional 'lost in the middle' effect for many models, but persistent biases related to relative positioning—especially in retrieval tasks—and reveal that model size alone does not fix these biases, highlighting the need for targeted mitigation and contextualization strategies. Overall, LongPiBench provides a rigorous framework for diagnosing and addressing positional biases in long-context LLMs, with implications for improving robustness in real-world long-text applications.

Abstract

Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. Thorough experiments are conducted with five commercial and six open-source models. These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases to advance LLM's capabilities.

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

TL;DR

This work investigates how the spacing and distribution of multiple relevant information pieces in long-context inputs bias LLMs. It introduces LongPiBench, a benchmark that isolates absolute and relative positional effects across four input lengths up to 256K tokens and evaluates nine models. The findings show reduced susceptibility to the traditional 'lost in the middle' effect for many models, but persistent biases related to relative positioning—especially in retrieval tasks—and reveal that model size alone does not fix these biases, highlighting the need for targeted mitigation and contextualization strategies. Overall, LongPiBench provides a rigorous framework for diagnosing and addressing positional biases in long-context LLMs, with implications for improving robustness in real-world long-text applications.

Abstract

Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. Thorough experiments are conducted with five commercial and six open-source models. These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases to advance LLM's capabilities.

Paper Structure

This paper contains 42 sections, 4 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Long-Context Benchmarks
  4. Long Context Data Augmentation
  5. LongPiBench
  6. Core Statistics
  7. Seed Data Annotation
  8. Data Augmentation
  9. Absolute Position.
  10. Relative Position.
  11. Experimental Setup
  12. Models.
  13. Metric.
  14. Context Length.
  15. Results and Discussion
  16. ...and 27 more sections

Figures (6)

  • Figure 1: Illustration of absolute position and relative position. Absolute position refers to the location of relevant information within the entire context sequence, while relative position represents the distribution and distance between multiple relevant information pieces.
  • Figure 2: Construction and task examples of LongPiBench. We manually annotated seed data and varied the positions of relevant information for data augmentation.
  • Figure 3: The impact of relevant information's absolute and relative position on Geimini-1.5-Flash geminiteam2024gemini15unlockingmultimodal, Claude-3.5-Haiku anthropic2024claude and Qwen 2.5 model family qwen2.5. A higher absolute position feature level indicates locations closer to the end of input, while a higher relative position feature level indicates a greater distance between relevant pieces of information.
  • Figure 4: Impact of query placement (beginning, end, both) on the performance of GPT-4o-mini openai2024gpt4o and Qwen-2.5-14B qwen2.5 models.
  • Figure 5: The impact of relevant information’s absolute and relative position for all open-source commercial models. A higher absolute position feature level indicates locations closer to the end of input, while a higher relative position feature level indicates a greater distance between relevant pieces of information.
  • ...and 1 more figures