A Comprehensive Study on the Use of Word Embedding Models in Software Engineering Domain

TL;DR

This paper addresses how word embedding models are used to semantically represent software engineering artifacts. It uses a systematic literature review of 181 primary studies across 45 CCf-ranked venues to map practice, model choices, training strategies, and evaluation practices. Findings show Word2Vec and BERT are predominant, with SE-specific embeddings favored; however, a large majority of studies lack direct comparisons with traditional semantic representations or across WE models. The results outline actionable directions, including expanding WE to less-explored SE tasks and artifacts, conducting rigorous comparative analyses, and developing domain-adapted or hybrid embeddings to improve SE automation.

Abstract

Word embedding (WE) techniques are advanced textual semantic representation models oriented from the natural language processing (NLP) area. Inspired by their effectiveness in facilitating various NLP tasks, more and more researchers attempt to adopt these WE models for their software engineering (SE) tasks, of which semantic representation of software artifacts such as bug reports and code snippets is the basis for further model building. However, existing studies are generally isolated from each other without comprehensive comparison and discussion. This not only makes the best practice of such cross-discipline technique adoption buried in scattered papers, but also makes us kind of blind to current progress in the semantic representation of SE artifacts. To this end, we decided to perform a comprehensive study on the use of WE models in the SE domain. 181 primary studies published in mainstream software engineering venues are collected for analysis. Several research questions related to the SE applications, the training strategy of WE models, the comparison with traditional semantic representation methods, etc., are answered. With the answers, we get a systematical view of the current practice of using WE for the SE domain, and figure out the challenges and actions in adopting or developing practical semantic representation approaches for the SE artifacts used in a series of SE tasks.

Figures (4)

• Figure 1: Primary Studies (#PS) by year and publication.
• Figure 2: The distribution of software artifacts that use WE models for semantic representation in all 181 papers.
• Figure 3: The distribution of different WE models in SE.
• Figure 4: The type and training methods of WE in SE.