Attention Overflow: Language Model Input Blur during Long-Context Missing Items Recommendation
Damien Sileo
TL;DR
This work investigates how LLMs struggle to identify a missing element in long, potentially repetitive inputs, a problem termed attention overflow. Formally, given a finite itemset $\mathcal{S}$ and a randomly removed element $y$, with $X=\mathcal{S}\setminus\{y\}$ scrambled in the prompt, the task is to recover $y$; the authors study this across synthetic numbers and MovieLens-based movie lists. They report that accuracy remains decent for small to moderate itemsets but degrades and repetition rises when $N$ grows beyond roughly $128$–$256$, indicating a fundamental limitation in attending to all input items simultaneously. Through zero-shot, fine-tuning, and contrastive evaluations, and with a public dataset supporting itemsets up to $8192$, they demonstrate both the practical impact on long-context conversational recommendations and the architectural challenges of current attention mechanisms, motivating caution in interpreting context lengths for generation tasks.
Abstract
Large language models (LLMs) can suggest missing elements from items listed in a prompt, which can be used for list completion or recommendations based on users' history. However, their performance degrades when presented with too many items, as they start to suggest items already included in the input list. This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this phenomenon on both synthetic problems (e.g., finding missing numbers in a given range of shuffled integers) and realistic movie recommendation scenarios. We refer to this issue as \textit{attention overflow}, as preventing repetition requires attending to all items simultaneously. Although iterative loops can mitigate this problem, their costs increase with the repetition rate, affecting the language models' ability to derive novelty from lengthy inputs.