Assessing Consensus of Developers' Views on Code Readability
Agnia Sergeyuk, Olga Lvova, Sergey Titov, Anastasiia Serova, Farid Bagirov, Timofey Bryksin
TL;DR
This study investigates whether Java developers with similar backgrounds share a consensus on Code Readability and its underlying aspects, addressing prior findings that readability models poorly reflect human judgments. Using 30 AI-generated Java snippets evaluated by 10 developers across 12 readability dimensions, the authors quantify agreement via ICC and associations with readability via Pearson r. They report moderate-to-good agreement on readability and significant correlations between all 12 aspects and readability, with Naming, Code Length, Understandable Goal, and Reading Flow showing the strongest links, suggesting stable targets for aligning LLM outputs with developer notions. The work implies that, within an organizational context, LLMs can be guided to produce more readable code by targeting robust readability cues, advancing Human-AI coding collaboration in the AI era.
Abstract
The rapid rise of Large Language Models (LLMs) has changed software development, with tools like Copilot, JetBrains AI Assistant, and others boosting developers' productivity. However, developers now spend more time reviewing code than writing it, highlighting the importance of Code Readability for code comprehension. Our previous research found that existing Code Readability models were inaccurate in representing developers' notions and revealed a low consensus among developers, highlighting a need for further investigations in this field. Building on this, we surveyed 10 Java developers with similar coding experience to evaluate their consensus on Code Readability assessments and related aspects. We found significant agreement among developers on Code Readability evaluations and identified specific code aspects strongly correlated with Code Readability. Overall, our study sheds light on Code Readability within LLM contexts, offering insights into how these models can align with developers' perceptions of Code Readability, enhancing software development in the AI era.