RAL-Bench: Benchmarking for Application-Level Functional Correctness and Non-Functional Quality Attributes

TL;DR

RAL-Bench tackles the challenge of generating runnable, multi-file repositories from natural-language requirements by grounding tasks in real-world GitHub projects, constructing black-box functional and non-functional tests, and filtering candidate tests against a reference implementation to ensure a sound oracle. It defines functional correctness as system-test pass rate and non-functional quality via five ISO/IEC 25010-inspired dimensions aggregated with an Analytic Hierarchy Process (AHP) weighting, yielding NF = 0.36 M + 0.24 S + 0.16 Rb + 0.12 E + 0.12 Ru, with per-dimension diagnostics and reference-based normalization. A comprehensive evaluation of 16 LLMs shows that functional correctness is the dominant bottleneck, with none exceeding 45% functional pass, while non-functional quality scores are higher but cannot compensate for functional gaps. The study reveals failure modes dominated by requirement–implementation mismatches and non-functional quality issues, questions the efficacy of mainstream one-shot generation strategies, and advocates a process-centric approach that emphasizes explicit functional requirements, cross-module repairs, and quality-aware generation for practical application-level code synthesis. The work provides a reproducible benchmark and a path toward more realistic, end-to-end evaluation of LLM-driven software creation in real-world contexts.

Abstract

Code generation has advanced rapidly with code-focused large language models (LLMs), especially on snippet-level tasks. However, application-level generation requires producing a runnable multi-file repository with correct structure, dependencies, and end-to-end executability, and real-world software must satisfy both functional correctness and non-functional quality (e.g., maintainability, security). Existing benchmarks provide a limited execution-based assessment of these requirements at the application level. We ask: Can current LLMs generate application-level repositories that meet both functional and non-functional criteria? We propose RAL-Bench, a benchmark and evaluation framework for application-level code generation. For each task, we distill a concise natural-language requirement from a high-quality reference project, build black-box system tests covering functional and non-functional attributes, and keep only tests that pass on the reference repository to ensure a sound oracle and an end-to-end executable suite. Functional correctness is measured by system-test pass rate. Non-functional quality is measured along five ISO/IEC 25010-inspired dimensions and aggregated with an Analytic Hierarchy Process (AHP)-derived weight vector, with per-dimension diagnostics and baseline-normalized scoring using reference measurements. Across 16 LLMs evaluated zero-shot with greedy decoding, functional correctness is the dominant bottleneck: no model exceeds a 45% functional pass rate under our requirement-driven, reference-validated tests. We release RAL-Bench at https://github.com/Wwstarry/RAL-Bench. .

Figures (6)

• Figure 1: Scope comparison between snippet-level and application-level code generation.
• Figure 2: Overall construction pipeline of RAL-Bench.
• Figure 3: Evaluation pipeline overview.
• Figure 4: Failure-pattern distribution for application-level code generation (left: breakdown by model; right: overall composition).
• Figure 5: Case studies of three representative failure modes in application-level code generation.