Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

Stefano Rando, Luca Romani, Alessio Sampieri, Luca Franco, John Yang, Yuta Kyuragi, Fabio Galasso, Tatsunori Hashimoto

TL;DR

LongCodeBench addresses the challenge of evaluating large-context coding LLMs by introducing tasks that require processing up to 1M tokens using real-world GitHub data. It combines LongCodeQA for code comprehension and LongSWE-Bench for patch-based debugging, providing multi-bracket evaluations and an execution-based metric for repairs. The study finds that long-context degrades performance across models and tasks, even among state-of-the-art systems, highlighting a gap between claimed context capacity and practical robustness. By releasing the dataset and code, LCB enables reproducible, real-world long-context benchmarking in software engineering.

Abstract

Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5. The LCB dataset is available publicly at https://huggingface.co/datasets/Steefano/LCB and the codebase to replicate the work on this paper at https://github.com/Zteefano/long-code-bench.

LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

TL;DR

LongCodeBench addresses the challenge of evaluating large-context coding LLMs by introducing tasks that require processing up to 1M tokens using real-world GitHub data. It combines LongCodeQA for code comprehension and LongSWE-Bench for patch-based debugging, providing multi-bracket evaluations and an execution-based metric for repairs. The study finds that long-context degrades performance across models and tasks, even among state-of-the-art systems, highlighting a gap between claimed context capacity and practical robustness. By releasing the dataset and code, LCB enables reproducible, real-world long-context benchmarking in software engineering.

Abstract

Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5. The LCB dataset is available publicly at https://huggingface.co/datasets/Steefano/LCB and the codebase to replicate the work on this paper at https://github.com/Zteefano/long-code-bench.
Paper Structure (24 sections, 6 figures, 6 tables)

This paper contains 24 sections, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Benchmarks for LCLMs
  4. Coding benchmarks for LCLMs
  5. LongCodeBench
  6. Code comprehension and repair
  7. LongCodeQA
  8. LongSWE-Bench
  9. Analysis
  10. Selection of LCLMs
  11. Performance on LongCodeQA
  12. Performance on LongSWE-Bench
  13. Discussion
  14. Conclusion
  15. Further analysis on data collection
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Increasing trend of LCLM context lengths over time. Models tested on LCB are highlighted in orange.
  • Figure 2: Input prompt structure and output format for the two LCB tasks. LongCodeQA (left) answer a multiple-choice question considering the full repository and the question derived from GitHub issues. LongSWE-Bench (right) generate a bug-fixing patch considering a subset of the codebase files and the GitHub issue.
  • Figure 3: Accuracy of Gemini 2 Flash on LongCodeQA split into two subsets: samples whose prompts have an average # Tokens/File below or above median.
  • Figure 4: Distribution of question topics in LongCodeQA, as inferred by an LLM (GPT-4o). The accuracy of Gemini 1.5 Pro for each topic subset is reported and shown as the respective slice radius.
  • Figure 5: Percentage of solved issues by Claude 3.5 Sonnet on a smaller version of LongSWE-Bench, grouped by bins of equal length range. The red line indicates a trend extracted through logistic regression, with a $p$-value of $0.005$—below the threshold ($0.01$) of statistical significance.
  • ...and 1 more figures