Quality Assurance of LLM-generated Code: Addressing Non-Functional Quality Characteristics

TL;DR

This study systematically investigates the non-functional quality characteristics (NFQCs) of LLM-generated code through a three-pronged approach: a systematic literature review mapped to ISO/IEC 25010, industry workshops capturing practitioner perspectives, and an empirical evaluation of NFQC trade-offs in generated patches using three LLMs. The findings reveal a mismatch among academic focus, industry priorities, and model performance: security and performance dominate research, while maintainability and readability are prioritized by practitioners but under-assessed; NFQC optimization often trades off against functional correctness. The empirical results show substantial variance across models and prompts, with generated patches typically lagging gold-standard patches in function, and NFQC-specific prompts sometimes improving one quality at the expense of others. The paper argues for integrating NFQC-aware verification, standardized evaluation frameworks, and tool-supported feedback into LLM-code pipelines to reduce technical debt and improve real-world reliability of AI-generated code.

Abstract

In recent years, LLMs have been widely integrated into software engineering workflows, supporting tasks like code generation. However, while these models often generate functionally correct outputs, we still lack a systematic understanding and evaluation of their non-functional qualities. Existing studies focus mainly on whether generated code passes the tests rather than whether it passes with quality. Guided by the ISO/IEC 25010 quality model, this study conducted three complementary investigations: a systematic review of 108 papers, two industry workshops with practitioners from multiple organizations, and an empirical analysis of patching real-world software issues using three LLMs. Motivated by insights from both the literature and practitioners, the empirical study examined the quality of generated patches on security, maintainability, and performance efficiency. Across the literature, we found that security and performance efficiency dominate academic attention, while maintainability and other qualities are understudied. In contrast, industry experts prioritize maintainability and readability, warning that generated code may accelerate the accumulation of technical debt. In our evaluation of functionally correct patches generated by three LLMs, improvements in one quality dimension often come at the cost of others. Runtime and memory results further show high variance across models and optimization strategies. Overall, our findings reveal a mismatch between academic focus, industry priorities, and model performance, highlighting the urgent need to integrate quality assurance mechanisms into LLM code generation pipelines to ensure that future generated code not only passes tests but truly passes with quality.

Figures (15)

• Figure 1: Search and selection process
• Figure 2: Overview of the experiment procedure. The workflow is divided into three main stages: (1) Baseline Evaluation, where initial patches were generated using SWE-agent and different LLMs, generated and benchmark gold patches are evaluated with Docker and CodeQL; (2) Filter and Prompt Design, where baseline results are analyzed and NFQC-specific prompts are constructed; and (3) NFQC-specific Regeneration and Evaluation, where patches are regenerated using NFQC-specific prompts, re-evaluated and compared against the baseline to analyze improvements and potential trade-offs.
• Figure 3: SWE-bench Lite Instance Structure
• Figure 4: SWE-agent Configurations
• Figure 5: The year distribution of papers.