Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs

Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bogomolov, Andrey Sokolov, Timofey Bryksin, Danny Dig

TL;DR

Problem: Extract Method refactoring guidance from automated tools often diverges from real developer practices. Approach: EM-Assist uses LLMs to generate candidate extractions, then validates and enhances them with static analysis and program slicing, and ranks the results before applying the selected refactoring inside IntelliJ. Contributions and findings: it achieves Recall@5 on 1,752 real refactorings that exceeds the previous best by a substantial margin, and a usability study with industry developers reports high acceptance. Significance: demonstrates that LLMs, when properly filtered and IDE-validated, can serve as practical, safe assistants for in-IDE refactoring and can be extended to additional languages and refactorings.

Abstract

Excessively long methods, loaded with multiple responsibilities, are challenging to understand, debug, reuse, and maintain. The solution lies in the widely recognized Extract Method refactoring. While the application of this refactoring is supported in modern IDEs, recommending which code fragments to extract has been the topic of many research tools. However, they often struggle to replicate real-world developer practices, resulting in recommendations that do not align with what a human developer would do in real life. To address this issue, we introduce EM-Assist, an IntelliJ IDEA plugin that uses LLMs to generate refactoring suggestions and subsequently validates, enhances, and ranks them. Finally, EM-Assist uses the IntelliJ IDE to apply the user-selected recommendation. In our extensive evaluation of 1,752 real-world refactorings that actually took place in open-source projects, EM-Assist's recall rate was 53.4% among its top-5 recommendations, compared to 39.4% for the previous best-in-class tool that relies solely on static analysis. Moreover, we conducted a usability survey with 18 industrial developers and 94.4% gave a positive rating.

EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs

TL;DR

Problem: Extract Method refactoring guidance from automated tools often diverges from real developer practices. Approach: EM-Assist uses LLMs to generate candidate extractions, then validates and enhances them with static analysis and program slicing, and ranks the results before applying the selected refactoring inside IntelliJ. Contributions and findings: it achieves Recall@5 on 1,752 real refactorings that exceeds the previous best by a substantial margin, and a usability study with industry developers reports high acceptance. Significance: demonstrates that LLMs, when properly filtered and IDE-validated, can serve as practical, safe assistants for in-IDE refactoring and can be extended to additional languages and refactorings.

Abstract

Excessively long methods, loaded with multiple responsibilities, are challenging to understand, debug, reuse, and maintain. The solution lies in the widely recognized Extract Method refactoring. While the application of this refactoring is supported in modern IDEs, recommending which code fragments to extract has been the topic of many research tools. However, they often struggle to replicate real-world developer practices, resulting in recommendations that do not align with what a human developer would do in real life. To address this issue, we introduce EM-Assist, an IntelliJ IDEA plugin that uses LLMs to generate refactoring suggestions and subsequently validates, enhances, and ranks them. Finally, EM-Assist uses the IntelliJ IDE to apply the user-selected recommendation. In our extensive evaluation of 1,752 real-world refactorings that actually took place in open-source projects, EM-Assist's recall rate was 53.4% among its top-5 recommendations, compared to 39.4% for the previous best-in-class tool that relies solely on static analysis. Moreover, we conducted a usability survey with 18 industrial developers and 94.4% gave a positive rating.
Paper Structure (11 sections, 3 figures)

This paper contains 11 sections, 3 figures.

Table of Contents

  1. introduction
  2. EM-Assist
  3. Example Usage of EM-Assist
  4. Workflow
  5. Implementation
  6. evaluation
  7. Effectiveness of EM-Assist
  8. Usability of EM-Assist
  9. related work
  10. Conclusions
  11. Acknowledgements

Figures (3)

  • Figure 1: Workflow for using EM-Assist within IntelliJ IDEA: 1) the user triggers the plugin to generate suggestions, 2) the plugin displays three refactoring options in a popup window, 3) the user selects one of the options and inspects the final code.
  • Figure 2: The workflow of generating refactoring suggestions
  • Figure 3: Results of the conducted usability survey.