Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs

TL;DR

This work analyzes positional bias in LLMs operating with long input contexts, showing that models exhibit distinct positional preferences rather than a universal 'lost-in-the-middle' effect. It introduces Position-Aware Parameter Efficient Fine-Tuning (PAPEFT), combining a data augmentation strategy that permutes candidate order with a lightweight Location Encoding adapter to make models attend uniformly across positions. Empirical results on recommendation and link-prediction tasks demonstrate substantial reductions in performance fluctuation ($\approx$54-59%) and meaningful accuracy gains ($\approx$57-64%), with the LE variant delivering additional improvements at far lower parameter cost than LoRA. The approach offers a practical, efficient path to robust long-context reasoning in retrieval-augmented settings and highlights the persistence of pre-training–induced biases that prompt further investigation into bias-aware fine-tuning methods.

Abstract

Recent advances in large language models (LLMs) have enhanced their ability to process long input contexts. This development is particularly crucial for tasks that involve retrieving knowledge from an external datastore, which can result in long inputs. However, recent studies show a positional bias in LLMs, demonstrating varying performance depending on the location of useful information within the input sequence. In this study, we conduct extensive experiments to investigate the root causes of positional bias. Our findings indicate that the primary contributor to LLM positional bias stems from the inherent positional preferences of different models. We demonstrate that merely employing prompt-based solutions is inadequate for overcoming the positional preferences. To address this positional bias issue of a pre-trained LLM, we developed a Position-Aware Parameter Efficient Fine-Tuning (PAPEFT) approach which is composed of a data augmentation technique and a parameter efficient adapter, enhancing a uniform attention distribution across the input context. Our experiments demonstrate that the proposed approach effectively reduces positional bias, improving LLMs' effectiveness in handling long context sequences for various tasks that require externally retrieved knowledge.

Figures (3)

• Figure 1: Illustration of Positional Preferences in LLMs: The figure demonstrates how the Vicuna-13b-v1.5-16k model's performance on a recommendation task changes with the correct answer's position in the input context window. Given a list of potential candidates, we intentionally position the ground truth candidate at various locations within the list to assess how the predicted position distribution by the LLM shifts. From the figure we can observe the probability peaks near the correct position of relevant information, demonstrating a degree of capacity for identifying pertinent information. There is a notable preference for the first position, indicating significant positional preference.
• Figure 2: The Longchat-13b-16k model's performance on a recommendation task changes with the correct answer's position in the input context window. Comparing with the Vicuna-13b-v1.5-16k model's trend in Figure \ref{['fig:prefrence']}, we can observe that these two models have different preferred positions. The Longchat-13b-16k model has preferred location around position eleven, while Vicuna-13b-v1.5-16k prefers the first position.
• Figure 3: The overall framework of location encoding soft prompt adapter module. The relative locations of potential documents are initially computed and subsequently fed into a soft prompt adapter. The soft location tokens are concatenated with textual tokens to form a combined input for the attention layers in LLMs.