Human-Aligned Code Readability Assessment with Large Language Models

TL;DR

CoReEval introduces a large-scale benchmark to study large language models as human-aligned code readability evaluators. By evaluating 10 LLMs across Java, Python, and CUDA on functional and test code with diverse prompting strategies and developer personas, the work probes alignment to human judgments and the quality of explanations. Results show that developer-guided prompts can improve justification quality and aspect coverage in structured contexts, though they may increase score variability, highlighting a trade-off between personalization and stability. The framework provides a reproducible foundation for prompt engineering, model alignment, and human-in-the-loop evaluation with broad applicability to education, onboarding, and CI/CD workflows.

Abstract

Code readability is crucial for software comprehension and maintenance, yet difficult to assess at scale. Traditional static metrics often fail to capture the subjective, context-sensitive nature of human judgments. Large Language Models (LLMs) offer a scalable alternative, but their behavior as readability evaluators remains underexplored. We introduce CoReEval, the first large-scale benchmark for evaluating LLM-based code readability assessment, comprising over 1.4 million model-snippet-prompt evaluations across 10 state of the art LLMs. The benchmark spans 3 programming languages (Java, Python, CUDA), 2 code types (functional code and unit tests), 4 prompting strategies (ZSL, FSL, CoT, ToT), 9 decoding settings, and developer-guided prompts tailored to junior and senior personas. We compare LLM outputs against human annotations and a validated static model, analyzing numerical alignment (MAE, Pearson's, Spearman's) and justification quality (sentiment, aspect coverage, semantic clustering). Our findings show that developer-guided prompting grounded in human-defined readability dimensions improves alignment in structured contexts, enhances explanation quality, and enables lightweight personalization through persona framing. However, increased score variability highlights trade-offs between alignment, stability, and interpretability. CoReEval provides a robust foundation for prompt engineering, model alignment studies, and human in the loop evaluation, with applications in education, onboarding, and CI/CD pipelines where LLMs can serve as explainable, adaptable reviewers.

Figures (4)

• Figure 1: Overview of our explorative study.
• Figure 2: Likert Distribution by Code Type (Java only).
• Figure 3: Sentiment Polarity of Justifications by GPT-4o and RoBERTa-TweetEval.
• Figure 4: Aspect mentions and reasoning structures in LLM justifications.