Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion

Dengfeng Liu, Jucai Zhai, Xiaoguang Jiang, Ziqun Li, Qianjin Yu, Feng Liu, Rui Ye, Huang Liu, Zhiguo Yang, Yongsheng Du, Fang Tan

TL;DR

This work addresses the misalignment between conventional code-completion metrics and user-perceived usefulness in on-the-fly editing. It introduces two user-aligned metrics, Longest Common Prefix (LCP) and ROUGE-LCP, and links them to probabilistic model outputs, supported by large-scale user data. To improve repository-level understanding, it proposes a Structure-Preserving and Semantically-Reordered Code Graph (SPSR-Graph) built from AST-based semantic units and cross-file dependencies, enabling structured pretraining. Empirical results show that LCP/ROUGE-LCP correlate more strongly with adoption than traditional metrics and that the SPSR-Graph corpus processing yields consistent gains in EM and BLEU, validating both the metrics and data framework for practical LLM-based code completion.

Abstract

Code completion technology based on large language model has significantly improved the development efficiency of programmers. However, in practical applications, there remains a gap between current commonly used code completion evaluation metrics and users' actual perception. To address this issue, we propose two evaluation metrics for code completion tasks--LCP and ROUGE-LCP, from the perspective of probabilistic modeling. Furthermore, to tackle the lack of effective structural semantic modeling and cross-module dependency information in LLMs for repository-level code completion scenarios, we propose a data processing method based on a Structure-Preserving and Semantically-Reordered Code Graph (SPSR-Graph). Through theoretical analysis and experimental validation, we demonstrate the superiority of the proposed evaluation metrics in terms of user perception consistency, as well as the effectiveness of the data processing method in enhancing model performance.

Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion

TL;DR

This work addresses the misalignment between conventional code-completion metrics and user-perceived usefulness in on-the-fly editing. It introduces two user-aligned metrics, Longest Common Prefix (LCP) and ROUGE-LCP, and links them to probabilistic model outputs, supported by large-scale user data. To improve repository-level understanding, it proposes a Structure-Preserving and Semantically-Reordered Code Graph (SPSR-Graph) built from AST-based semantic units and cross-file dependencies, enabling structured pretraining. Empirical results show that LCP/ROUGE-LCP correlate more strongly with adoption than traditional metrics and that the SPSR-Graph corpus processing yields consistent gains in EM and BLEU, validating both the metrics and data framework for practical LLM-based code completion.

Abstract

Code completion technology based on large language model has significantly improved the development efficiency of programmers. However, in practical applications, there remains a gap between current commonly used code completion evaluation metrics and users' actual perception. To address this issue, we propose two evaluation metrics for code completion tasks--LCP and ROUGE-LCP, from the perspective of probabilistic modeling. Furthermore, to tackle the lack of effective structural semantic modeling and cross-module dependency information in LLMs for repository-level code completion scenarios, we propose a data processing method based on a Structure-Preserving and Semantically-Reordered Code Graph (SPSR-Graph). Through theoretical analysis and experimental validation, we demonstrate the superiority of the proposed evaluation metrics in terms of user perception consistency, as well as the effectiveness of the data processing method in enhancing model performance.
Paper Structure (28 sections, 8 equations, 8 figures, 3 tables, 2 algorithms)

This paper contains 28 sections, 8 equations, 8 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation Metrics for Code Completion Tasks
  4. Repository-Level Code Completion
  5. Method
  6. User Behavior Analysis
  7. Design of LCP and ROUGE-LCP Metrics
  8. Repository-Level Code Corpus Processing Framework
  9. Syntax-Aware Semantic Unit Extraction via AST
  10. Structure-Preserving and Semantically-Reordered Code Graph
  11. Experiments
  12. Experimental Setup
  13. LCP and ROUGE-LCP Distribution and Correlation with Adoption Rate
  14. Comparison with General Metrics
  15. Impact of Different Pretraining Corpus Strategies
  16. ...and 13 more sections

Figures (8)

  • Figure 1: Overview of the proposed framework. The framework includes three stages: corpus preprocessing, AST structural segmentation, and structure-aware graph construction, transitioning from local semantic preservation to global dependency modeling to systematically construct structure-aware code completion corpora.
  • Figure 2: LCP Distribution and Its Relationship with Adoption Count and Adoption Rate
  • Figure 3: ROUGE-LCP Distribution and Its Relationship with Adoption Count and Adoption Rate
  • Figure 4: Daily Metric and Adoption Rate Distributions, Heatmap of Correlation Between Evaluation Metrics and Adoption Rate. R-L refers to ROUGE-L, R-LCP refers to ROUGE-LCP, and AR refers to Adoption Rate.
  • Figure 5: Impact of code knowledge graph breadth on code completion performance
  • ...and 3 more figures