RepoQA: Evaluating Long Context Code Understanding
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, Lingming Zhang
TL;DR
RepoQA tackles long-context code understanding by moving beyond retrieval to assess whether LLMs can locate and reproduce functions from large code bases given natural-language descriptions. It defines the Searching Needle Function task, builds a 500-task multilingual dataset from 50 repositories across 5 languages, and provides an automatic pipeline for data curation and evaluation. The study across 33 models shows a gap between open-source and proprietary systems, with language and model family affecting performance, and reveals that removing comments can sometimes improve understanding. The benchmark and findings point to the need for longer-context reasoning capabilities in code-related AI and offer directions to extend SNF with more languages and tasks.
Abstract
Recent advances have been improving the context windows of Large Language Models (LLMs). To quantify the real long-context capabilities of LLMs, evaluators such as the popular Needle in a Haystack have been developed to test LLMs over a large chunk of raw texts. While effective, current evaluations overlook the insight of how LLMs work with long-context code, i.e., repositories. To this end, we initiate the RepoQA benchmark to evaluate LLMs on long-context code understanding. Traditional needle testers ask LLMs to directly retrieve the answer from the context without necessary deep understanding. In RepoQA, we built our initial task, namely Searching Needle Function (SNF), which exercises LLMs to search functions given their natural-language description, i.e., LLMs cannot find the desired function if they cannot understand the description and code. RepoQA is multilingual and comprehensive: it includes 500 code search tasks gathered from 50 popular repositories across 5 modern programming languages. By evaluating 26 general and code-specific LLMs on RepoQA, we show (i) there is still a small gap between the best open and proprietary models; (ii) different models are good at different languages; and (iii) models may understand code better without comments.