Measuring how changes in code readability attributes affect code quality evaluation by Large Language Models

TL;DR

This work investigates whether large-language models (LLMs) can generate semantic, reproducible metrics for code readability by varying key readability attributes. It uses a quasi-experimental design with nine LLMs and three targeted interventions on 12 Java classes, benchmarking against SonarQube and Scalabrino's Classifier. Findings show LLMs are sensitive to identifier naming and code smells, with high agreement to SCO on original and refactored code but notable semantic sensitivity and variability across interventions. The results suggest LLMs hold potential as a complementary, semantics-aware readability assessment, while highlighting the need for human validation and broader language coverage to ensure practical applicability.

Abstract

Code readability is one of the main aspects of code quality, influenced by various properties like identifier names, comments, code structure, and adherence to standards. However, measuring this attribute poses challenges in both industry and academia. While static analysis tools assess attributes such as code smells and comment percentage, code reviews introduce an element of subjectivity. This paper explores using Large Language Models (LLMs) to evaluate code quality attributes related to its readability in a standardized, reproducible, and consistent manner. We conducted a quasi-experiment study to measure the effects of code changes on Large Language Model (LLM)s interpretation regarding its readability quality attribute. Nine LLMs were tested, undergoing three interventions: removing comments, replacing identifier names with obscure names, and refactoring to remove code smells. Each intervention involved 10 batch analyses per LLM, collecting data on response variability. We compared the results with a known reference model and tool. The results showed that all LLMs were sensitive to the interventions, with agreement with the reference classifier being high for the original and refactored code scenarios. The LLMs demonstrated a strong semantic sensitivity that the reference model did not fully capture. A thematic analysis of the LLMs reasoning confirmed their evaluations directly reflected the nature of each intervention. The models also exhibited response variability, with 9.37% to 14.58% of executions showing a standard deviation greater than zero, indicating response oscillation, though this did not always compromise the statistical significance of the results. LLMs demonstrated potential for evaluating semantic quality aspects, such as coherence between identifier names, comments, and documentation with code purpose.