Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?

Ziyuan He, Yuxuan Wang, Jiaqi Li, Kexin Liang, Muhan Zhang

TL;DR

LooGLE v2 introduces a scalable, real-world long-context benchmark spanning law, finance, game, and code to assess LLM long-dependency understanding across 10 domain-specific tasks (1,934 QA instances). Evaluations on 6 locally deployed and 4 API-based models reveal that even the strongest models (GPT-4.1) achieve only 59.2% on average, highlighting a substantial gap between extended context windows and practical long-context reasoning. The study shows that longer context alone often fails to improve performance, while chain-of-thought prompting yields mixed benefits and retrieval augmentation generally hurts unless in niche finance cases. These results underscore the need for architectural and methodological advances to enable robust long-context understanding in real-world tasks and provide a realistic evaluation framework to guide future research.

Abstract

Large language models (LLMs) are equipped with increasingly extended context windows recently, yet their long context understanding capabilities over long dependency tasks remain fundamentally limited and underexplored. This gap is especially significant in many real-world long-context applications that were rarely benchmarked. In this paper, we introduce LooGLE v2, a novel benchmark designed to evaluate LLMs' long context ability in real-world applications and scenarios. Our benchmark consists of automatically collected real-world long texts, ranging from 16k to 2M tokens, encompassing domains in law, finance, game and code. Accordingly, we delicately design 10 types of domain-specific long-dependency tasks and generate 1,934 QA instances with various diversity and complexity in a scalable data curation pipeline for further practical needs. We conduct a comprehensive assessment of 6 locally deployed and 4 API-based LLMs. The evaluation results show that even the best-performing model achieves only a 59.2% overall score on our benchmark. Despite the extensive context windows, popular LLMs are only capable of understanding a much shorter length of context than they claim to be, revealing significant limitations in their ability to handle real-world tasks with long dependencies and highlighting substantial room for model improvement in practical long-context understanding.

LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?

TL;DR

LooGLE v2 introduces a scalable, real-world long-context benchmark spanning law, finance, game, and code to assess LLM long-dependency understanding across 10 domain-specific tasks (1,934 QA instances). Evaluations on 6 locally deployed and 4 API-based models reveal that even the strongest models (GPT-4.1) achieve only 59.2% on average, highlighting a substantial gap between extended context windows and practical long-context reasoning. The study shows that longer context alone often fails to improve performance, while chain-of-thought prompting yields mixed benefits and retrieval augmentation generally hurts unless in niche finance cases. These results underscore the need for architectural and methodological advances to enable robust long-context understanding in real-world tasks and provide a realistic evaluation framework to guide future research.

Abstract

Large language models (LLMs) are equipped with increasingly extended context windows recently, yet their long context understanding capabilities over long dependency tasks remain fundamentally limited and underexplored. This gap is especially significant in many real-world long-context applications that were rarely benchmarked. In this paper, we introduce LooGLE v2, a novel benchmark designed to evaluate LLMs' long context ability in real-world applications and scenarios. Our benchmark consists of automatically collected real-world long texts, ranging from 16k to 2M tokens, encompassing domains in law, finance, game and code. Accordingly, we delicately design 10 types of domain-specific long-dependency tasks and generate 1,934 QA instances with various diversity and complexity in a scalable data curation pipeline for further practical needs. We conduct a comprehensive assessment of 6 locally deployed and 4 API-based LLMs. The evaluation results show that even the best-performing model achieves only a 59.2% overall score on our benchmark. Despite the extensive context windows, popular LLMs are only capable of understanding a much shorter length of context than they claim to be, revealing significant limitations in their ability to handle real-world tasks with long dependencies and highlighting substantial room for model improvement in practical long-context understanding.
Paper Structure (59 sections, 1 equation, 6 figures, 14 tables)

This paper contains 59 sections, 1 equation, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Our Benchmark
  4. Task design
  5. Data Collection and Task Curation
  6. Benchmarking LLMs on LooGLE v2
  7. Experimental Setup
  8. Experimental Results
  9. Main Results
  10. Performance on the Varying Input Lengths
  11. Performance with Chain-of-thoughts
  12. Performance on Domain-Specific Models
  13. Performance with Retrieve Based Methods
  14. Conclusions
  15. Acknowledgements
  16. ...and 44 more sections

Figures (6)

  • Figure 1: Overview dataset statistics for LooGLE v2
  • Figure 2: Overview description of LooGLE v2 tasks.
  • Figure 3: The overall model performance across different tasks on LooGLE v2 (Left: Raw, Right: Normalized to the highest score per task)
  • Figure 4: Comparison of model performance across varying input context lengths.
  • Figure 5: Comparison of model performance using RAG.
  • ...and 1 more figures