Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?

TL;DR

This study investigates the effectiveness of deep learning methods for identifying inconsistent method names, revealing that prior evaluations on balanced datasets do not translate well to real-world, imbalanced settings. A large benchmark, BenMark, combining automatic commit-history signals with manual inspection, shows substantial performance drops for state-of-the-art IR- and generation-based approaches when tested on natural data. Analyses attribute failures to weak method-body representations, naive similarity measures, and semantic gaps; IR-based methods excel on simple bodies and popular first-name tokens, while generation-based methods struggle with longer, more complex bodies and narrow semantic mismatches. The authors propose contrastive learning and leveraging large language models to address these weaknesses and highlight the need for realistic benchmarks and multi-faceted evaluation for practical deployment in software engineering tooling.

Abstract

Concise and meaningful method names are crucial for program comprehension and maintenance. However, method names may become inconsistent with their corresponding implementations, causing confusion and errors. Several deep learning (DL)-based approaches have been proposed to identify such inconsistencies, with initial evaluations showing promising results. However, these evaluations typically use a balanced dataset, where the number of inconsistent and consistent names are equal. This setup, along with flawed dataset construction, leads to false positives, making reported performance less reliable in real-world scenarios, where most method names are consistent. In this paper, we present an empirical study that evaluates state-of-the-art DL-based methods for identifying inconsistent method names. We create a new benchmark by combining automatic identification from commit histories and manual developer inspections, reducing false positives. We evaluate five representative DL approaches (one retrieval-based and four generation-based) on this benchmark. Our results show that performance drops substantially when moving from the balanced dataset to the new benchmark. We further conduct quantitative and qualitative analyses to understand the strengths and weaknesses of the approaches. Retrieval-based methods perform well on simple methods and those with popular name sub-tokens but fail due to inefficient representation techniques. Generation-based methods struggle with inaccurate similarity calculations and immature name generation. Based on these findings, we propose improvements using contrastive learning and large language models (LLMs). Our study suggests that significant improvements are needed before these DL approaches can be effectively applied to real-world software systems.

Figures (5)

• Figure 1: Popularity of F-tokens vs. Success Rate ($IRMCC$).
• Figure 2: Distribution of Methods Similarities.
• Figure 3: LOC VS. Success Rate.
• Figure 4: Complementarity of Five Selected Approaches in Identifying Inconsistent Method Names.
• Figure 5: Overview of the Contrastive Pre-training.