Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks

TL;DR

This work investigates whether replacing human-written pre-training comments with LLM-generated ones can improve code intelligence models. It introduces two reference-free evaluation tasks, Code Comment Inconsistency Detection and Semantic Code Search, to assess code-comment quality without gold references, finding that LLM-generated comments align more semantically with code. Building on this, the authors reconstruct CodeSearchNet using LLM-generated comments and pre-train CodeT5 on the rebuilt data, observing improvements in code summarization, NL-to-code generation, and translation, while code-centric tasks show varied gains. The results challenge the predominance of human references in datasets and highlight dataset quality as a lever for advancing code intelligence, with practical implications for constructing higher-quality pre-training corpora.

Abstract

Pre-trained code models rely heavily on high-quality pre-training data, particularly human-written reference comments that bridge code and natural language. However, these comments often become outdated as software evolves, degrading model performance. Large language models (LLMs) excel at generating high-quality code comments. We investigate whether replacing human-written comments with LLM-generated ones improves pre-training datasets. Since standard metrics cannot assess reference comment quality, we propose two novel reference-free evaluation tasks: code-comment inconsistency detection and semantic code search. Results show that LLM-generated comments are more semantically consistent with code than human-written ones, as confirmed by manual evaluation. Leveraging this finding, we rebuild the CodeSearchNet dataset with LLM-generated comments and re-pre-train CodeT5. Evaluations demonstrate that models trained on LLM-enhanced data outperform those using original human comments in code summarization, generation, and translation tasks. This work validates rebuilding pre-training datasets with LLMs to advance code intelligence, challenging the traditional reliance on human reference comments.

Figures (3)

• Figure 1: Code comment inconsistency detection as a reference-free evaluation metric.
• Figure 2: Code search as a reference-free evaluation metric.
• Figure 3: A Concode data sample rebuilt by GPT-3.5-Turbo.