Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents

Jing Gong, Yanghui Wu, Linxi Liang, Yanlin Wang, Jiachi Chen, Mingwei Liu, Zibin Zheng

TL;DR

CoSQA+ introduces a large-scale, multi-choice code search benchmark that pairs real user queries with multiple functionally matching codes and relies on a novel test-driven annotation pipeline. By combining multi-model candidate selection with automated test generation, execution, and arbitration, the approach achieves high labeling accuracy (0.939) and demonstrates superior training utility over CoSQA. The benchmark supports robust evaluation via MAP and cross-language generalization, with CodeBERT performing strongly on multi-choice tasks. The work highlights practical benefits and challenges of functionally verified, multi-example code search in real-world software development, and provides publicly available data to advance research in this area.

Abstract

Semantic code search, retrieving code that matches a given natural language query, is an important task to improve productivity in software engineering. Existing code search datasets face limitations: they rely on human annotators who assess code primarily through semantic understanding rather than functional verification, leading to potential inaccuracies and scalability issues. Additionally, current evaluation metrics often overlook the multi-choice nature of code search. This paper introduces CoSQA+, pairing high-quality queries from CoSQA with multiple suitable codes. We develop an automated pipeline featuring multiple model-based candidate selections and the novel test-driven agent annotation system. Among a single Large Language Model (LLM) annotator and Python expert annotators (without test-based verification), agents leverage test-based verification and achieve the highest accuracy of 93.9%. Through extensive experiments, CoSQA+ has demonstrated superior quality over CoSQA. Models trained on CoSQA+ exhibit improved performance. We publicly release both CoSQA+_all, which contains 412,080 agent-annotated pairs, and CoSQA+_verified, which contains 1,000 human-verified pairs, at https://github.com/DeepSoftwareAnalytics/CoSQA_Plus.

CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents

TL;DR

CoSQA+ introduces a large-scale, multi-choice code search benchmark that pairs real user queries with multiple functionally matching codes and relies on a novel test-driven annotation pipeline. By combining multi-model candidate selection with automated test generation, execution, and arbitration, the approach achieves high labeling accuracy (0.939) and demonstrates superior training utility over CoSQA. The benchmark supports robust evaluation via MAP and cross-language generalization, with CodeBERT performing strongly on multi-choice tasks. The work highlights practical benefits and challenges of functionally verified, multi-example code search in real-world software development, and provides publicly available data to advance research in this area.

Abstract

Semantic code search, retrieving code that matches a given natural language query, is an important task to improve productivity in software engineering. Existing code search datasets face limitations: they rely on human annotators who assess code primarily through semantic understanding rather than functional verification, leading to potential inaccuracies and scalability issues. Additionally, current evaluation metrics often overlook the multi-choice nature of code search. This paper introduces CoSQA+, pairing high-quality queries from CoSQA with multiple suitable codes. We develop an automated pipeline featuring multiple model-based candidate selections and the novel test-driven agent annotation system. Among a single Large Language Model (LLM) annotator and Python expert annotators (without test-based verification), agents leverage test-based verification and achieve the highest accuracy of 93.9%. Through extensive experiments, CoSQA+ has demonstrated superior quality over CoSQA. Models trained on CoSQA+ exhibit improved performance. We publicly release both CoSQA+_all, which contains 412,080 agent-annotated pairs, and CoSQA+_verified, which contains 1,000 human-verified pairs, at https://github.com/DeepSoftwareAnalytics/CoSQA_Plus.
Paper Structure (57 sections, 5 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 57 sections, 5 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation Survey
  3. Methodology
  4. Results and Analysis
  5. Related Work
  6. Code Search Datasets
  7. Code Search Methods
  8. Test-Driven Agents
  9. CoSQA+
  10. Query and Code Collection
  11. Query Source
  12. Code Source
  13. Candidate Pairs Construction
  14. Construction Steps
  15. Multi-model Embedding
  16. ...and 42 more sections

Figures (7)

  • Figure 1: An example of multi-choice code search. The query is vague in its definition of "illegal character," allowing eight candidate code snippets to match under different interpretations.
  • Figure 2: The construction process of CoSQA+. (a) Queries sourced from CoSQA and code snippets filtered from CodeSearchNet. (b) Candidate pairs selected based on query-code cosine similarity, computed using a multi-model approach. (c) Pipeline for annotation by test-driven agents.
  • Figure 3: Query counts grouped by number of matched codes
  • Figure 4: Probability of identifying matched codes as the number of candidate codes increases
  • Figure 5: Example illustration of (1) clear match, (2) clear non-match, and (3) unclear cases.
  • ...and 2 more figures