Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction

TL;DR

This work tackles why fine-tuned LLMs struggle to extract knowledge described later in training documents, a phenomenon tied to autoregressive training and termed a perplexity curse. By introducing synthetic Bio and Wiki2023+ datasets with QA pairs annotated to source sentences, the authors study how the position of the answer within a document affects extraction. They evaluate several training recipes, notably denoising auto-regressive training ($D$-AR), attention dropout, and sentence shuffling, demonstrating that vanilla autoregressive training degrades performance for middle/end positions while $D$-AR substantially improves extraction across positions and larger models with regularization close the gap. The findings inform the trade-offs between retrieval-augmented generation (RAG) and fine-tuning for domain adaptation and provide practical datasets to advance research on extractable knowledge in LLMs.

Abstract

Large language models require updates to remain up-to-date or adapt to new domains by fine-tuning them with new documents. One key is memorizing the latest information in a way that the memorized information is extractable with a query prompt. However, LLMs suffer from a phenomenon called perplexity curse; despite minimizing document perplexity during fine-tuning, LLMs struggle to extract information through a prompt sentence. In this new knowledge acquisition and extraction, we find a very intriguing fact that LLMs can accurately answer questions about the first sentence, but they struggle to extract information described in the middle or end of the documents used for fine-tuning. Our study suggests that the auto-regressive training causes this issue; each token is prompted by reliance on all previous tokens, which hinders the model from recalling information from training documents by question prompts. To conduct the in-depth study, we publish both synthetic and real datasets, enabling the evaluation of the QA performance w.r.t. the position of the corresponding answer in a document. Our investigation shows that even a large model suffers from the perplexity curse, but regularization such as denoising auto-regressive loss can enhance the information extraction from diverse positions. These findings will be (i) a key to improving knowledge extraction from LLMs and (ii) new elements to discuss the trade-off between RAG and fine-tuning in adapting LLMs to a new domain.

Figures (12)

• Figure 1: Left: In our problem setting, we fine-tune a pre-trained LLM on new documents and QA data created from the documents, and evaluate on hold-out QA data (green). Right: We vary the position of the answer sentence in training documents and fine-tune models on the modulated documents. Each plot shows a model tuned on documents with different answer positions. We find that the fine-tuned LLM suffers from the positional bias issue, i.e., it cannot accurately answer questions described in the middle or the end of documents. See Sec. \ref{['sec:answer_position']} for details.
• Figure 2: Visualization of studied four training methods. From left to right, (1) AR: standard auto-regressive training, (2) D-AR: denoising auto-regressive training randomly replaces input tokens with random ones while keeping the prediction target, (3) Shuffle: sentence shuffling shuffles input sentences, (4) Attn Drop: attention dropout randomly drops the attention in the self-attention module.
• Figure 3: We vary the position of the sentence in training documents and fine-tune a model on the modulated documents. The X-axis represents $k$, the position of the sentence corresponding to the answer in the training documents ($\bm{D}^{k}$). Each point corresponds to a model trained on $\bm{D}^{k}$. Left: EM accuracy w.r.t. person's birthday for bioS. We vary the position of the sentence describing a person's birthday. Right: EM accuracy on Wiki2023+. The position of the sentence corresponding to the answer varies from the 1st to the 5th.
• Figure 4: (a): Results of AR model with different sizes in modulating answer positions. We compare Llama-7B, 13B, and 70B for AR models. Models with increased parameter size improve performance, yet still suffer from the positional bias issue. (b)(c)(d): AR vs. D-AR in different model sizes. D-AR significantly improves performance over AR for all sizes. 70B model with D-AR greatly mitigates the effect from the answer position.
• Figure 5: Left: Analysis of perplexity conducted on models in Fig. \ref{['fig:em_wiki']}. We compute the perplexity for the first sentence in the original document ($\bm{s_1}$ in Sec.\ref{['sec:answer_position']}). The perplexity increases by putting the sentence latter for all models. AR model shows the highest perplexity. Middle: Perplexity comparison by model size. The smaller model relies more on the previous sentences to memorize a sentence. Right: Longer training benefits the performance improvements near the beginning of documents. Both analyses are conducted on Wiki2023+.