Personalized Code Readability Assessment: Are We There Yet?

TL;DR

The paper investigates whether code readability, traditionally treated as an objective property of code, can be personalized to individual developers. It compares generalist readability models against a GPT-4o-based personalized approach using two public datasets and few-shot learning, finding that personalization offers no clear advantage and is hampered by inconsistent annotations in existing datasets. The study emphasizes the need for more reliable, principled readability datasets and suggests that current generalist feature-based models remain the stronger baseline for developer-specific judgments. Overall, it calls for continued exploration of dataset quality and potential fine-tuning of models to better capture individual readers' notions of readability, with practical implications for maintenance prioritization and code review tooling.

Abstract

Unreadable code could be a breeding ground for errors. Thus, previous work defined approaches based on machine learning to automatically assess code readability that can warn developers when some code artifacts (e.g., classes) become unreadable. Given datasets of code snippets manually evaluated by several developers in terms of their perceived readability, such approaches (i) establish a snippet-level ground truth, and (ii) train a binary (readable/unreadable) or a ternary (readable/neutral/unreadable) code readability classifier. Given this procedure, all existing approaches neglect the subjectiveness of code readability, i.e., the possible different developer-specific nuances in the code readability perception. In this paper, we aim to understand to what extent it is possible to assess code readability as subjectively perceived by developers through a personalized code readability assessment approach. This problem is significantly more challenging than the snippet-level classification problem: We assume that, in a realistic scenario, a given developer is keen to provide only a few code readability evaluations, thus less data is available. For this reason, we adopt an LLM with few-shot learning to achieve our goal. Our results, however, show that such an approach achieves worse results than a state-of-the-art feature-based model that is trained to work at the snippet-level. We tried to understand why this happens by looking more closely at the quality of the available code readability datasets and assessed the consistency of the inter-developer evaluations. We observed that up to a third of the evaluations are self-contradictory. Our negative results call for new and more reliable code readability datasets.

Figures (4)

• Figure 1: Example of controversial snippet.
• Figure 2: F1-score boxplot for unreadable on $D_{s}$.
• Figure 3: F1-score boxplot for unreadable$D_{b\&w}$.
• Figure 4: Example of inconsistent evaluation from $D_{b\&w}$.