Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi, Matthew Purver

TL;DR

The paper reveals that even with very large context windows, LLMs struggle to analyze long sequences in sentiment analysis and news categorization. It introduces two input-condensing pipelines—Pure Extractive Summarization and Diverse Summarization—along with seven prompting strategies, evaluated across three long-form datasets and multiple models. Empirical results show performance gains up to about 50% and substantial reductions in API cost and latency, driven by selective truncation and diverse summarization. The work highlights the importance of input optimization for long-form NLP and motivates future research on specialized datasets and robust prompting techniques.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and analyzing lengthy sequential inputs, owing to their extensive context windows that allow processing millions of tokens in a single forward pass. However, this paper uncovers a surprising limitation: LLMs fall short when handling long input sequences. We investigate this issue using three datasets and two tasks (sentiment analysis and news categorization) across various LLMs, including Claude 3, Gemini Pro, GPT 3.5 Turbo, Llama 3 Instruct, and Mistral Instruct models. To address this limitation, we propose and evaluate ad-hoc solutions that substantially enhance LLMs' performance on long input sequences by up to 50%, while reducing API cost and latency by up to 93% and 50%, respectively.

Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

TL;DR

The paper reveals that even with very large context windows, LLMs struggle to analyze long sequences in sentiment analysis and news categorization. It introduces two input-condensing pipelines—Pure Extractive Summarization and Diverse Summarization—along with seven prompting strategies, evaluated across three long-form datasets and multiple models. Empirical results show performance gains up to about 50% and substantial reductions in API cost and latency, driven by selective truncation and diverse summarization. The work highlights the importance of input optimization for long-form NLP and motivates future research on specialized datasets and robust prompting techniques.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and analyzing lengthy sequential inputs, owing to their extensive context windows that allow processing millions of tokens in a single forward pass. However, this paper uncovers a surprising limitation: LLMs fall short when handling long input sequences. We investigate this issue using three datasets and two tasks (sentiment analysis and news categorization) across various LLMs, including Claude 3, Gemini Pro, GPT 3.5 Turbo, Llama 3 Instruct, and Mistral Instruct models. To address this limitation, we propose and evaluate ad-hoc solutions that substantially enhance LLMs' performance on long input sequences by up to 50%, while reducing API cost and latency by up to 93% and 50%, respectively.
Paper Structure (34 sections, 5 figures, 7 tables)

This paper contains 34 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Motivation.
  4. Contribution.
  5. Methodology
  6. Summarization Methodology
  7. Pure Extractive Summarization:
  8. Diverse Summarization:
  9. Prompting Scenarios
  10. Evaluation
  11. Datasets
  12. GameSpot Reviews GameSpotReviews:
  13. 20 Newsgroups 20NewsGroupDataset:
  14. BBC News Archive BBCNewsDataset:
  15. Experiments
  16. ...and 19 more sections

Figures (5)

  • Figure 1: The summarization pipelines for summarising information. The diverse summarization approach builds on top of the purely extractive approach but gives higher priority to lexical diversity.
  • Figure 2: Ablation study on the length of the selected truncation/summary for different scenarios using Claude 3 Haiku over 5 runs with 85% Confidence Intervals. The results show the efficacy of approaches optimizing LLMs' input. 'Full' context performs poorly on all metrics. Additionally, after the length of input exceeds 10 sentences, less meaningful improvement in the performance of all scenarios is observed. A similar trend is seen for all LLMs.
  • Figure 3: Performance gain (% accuracy boost) vs. normalized ARP score for each summarization/truncation scenario compared to the 'Full' baseline. Lower ARP scores (more cohesive corpora) generally yield higher performance gains across scenarios.
  • Figure 4: Ablation Study on temperature in the news categorization task on 20 newsgroup dataset. The results show the performance of the models does not experience much difference as we change the temperature from 0 to 0.1 aside from a slight increase in the confidence interval. These results are over 5 runs. The width of the 85% Confidence Interval for the 'Random Sampling' scenario is much bigger due to the randomness introduced by selecting the sentences.
  • Figure 5: LLMs performance over 5 runs on sentiment analysis on GameSpot dataset. The "Full Text" scenario is a horizontal line since it always contains the full text, not summary/truncated text and we have included it as a horizontal line here as baseline