LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding

TL;DR

LONGCODEU addresses the lack of a rigorous evaluation framework for long-context code understanding by introducing a comprehensive benchmark built from real-world Python repositories, encompassing four understanding aspects (code unit perception, intra-code unit understanding, inter-code unit relations, and long documentation understanding) across eight tasks and up to $128K$ tokens. The study evaluates nine LCLMs (six general and three code-focused) and finds substantial performance degradation once long code surpasses $32K$ tokens, with inter-code unit relations being the most challenging; code models generally outperform general models, though even the strongest models struggle at repository scale. The work provides practical guidance for model selection in software engineering tasks and highlights directions for improving long-code reasoning, such as disambiguating similar code units and distinguishing true understanding from memorization. Overall, LONGCODEU offers a realistic, multi-faceted framework to drive progress in long-context code understanding for real-world applications.

Abstract

Current advanced long-context language models offer great potential for real-world software engineering applications. However, progress in this critical domain remains hampered by a fundamental limitation: the absence of a rigorous evaluation framework for long code understanding. To gap this obstacle, we propose a long code understanding benchmark LONGCODEU from four aspects (8 tasks) to evaluate LCLMs' long code understanding ability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6 general models and 3 code models). Our experimental results reveal key limitations in current LCLMs' capabilities for long code understanding. Particularly, the performance of LCLMs drops dramatically when the long code length is greater than 32K, falling far short of their claimed 128K-1M context windows. In the four aspects, inter-code unit relation understanding is the most challenging for LCLMs. Our study provides valuable insights for optimizing LCLMs and driving advancements in software engineering.

Figures (7)

• Figure 1: Four understanding aspects in LongCodeU.
• Figure 2: Performance comparison across tasks and long code lengths on LongCodeU (grey blocks indicate unavailable configurations). The rate of performance degradation exhibits task-specific and model-specific patterns.
• Figure 3: Assessing long code understanding vs. memorization on CU_SA (left) and DRA_T2 (right) tasks.
• Figure 4: The value of Kendall-Tau $\tau$ between our automatic metrics and human evaluation.
• Figure 5: Performance comparison across long code lengths on tasks which can be measured by precision-based metrics. (grey blocks indicate unavailable configurations). The rate of performance degradation exhibits task-specific and model-specific patterns.