Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insights

Ahilan Ayyachamy Nadar Ponnusamy

TL;DR

This paper tackles the challenge of evaluating AI-generated code by introducing a reverse-generation framework that infers requirements from code and compares them to the original specifications using the SBC score. SBC combines semantic similarity (major weight), BLEU, and completeness to assess requirement-code alignment without relying on reference implementations. Through a dataset of 90 multi-layer requirements and four open LLMs, the study demonstrates stable performance trends across models and showcases how missing or extra elements can reveal hallucinations and gaps in generated code. The proposed approach offers actionable developer insights and can be integrated into AI code assistants to improve transparency and reliability in the software development lifecycle.

Abstract

The rise of Large Language Models (LLMs) in software engineering, particularly in code generation, has garnered significant attention. However, assessing the quality of AI-generated code remains a challenge due to the inherent complexity of programming tasks and the lack of robust evaluation metrics that align well with human judgment. Traditional token-based metrics such as BLEU and ROUGE, while commonly used in natural language processing, exhibit weak correlations with human assessments in code intelligence and verification tasks. Furthermore, these metrics are primarily research focused and are not designed for seamless integration into the software development lifecycle, limiting their practical utility for developers seeking to improve code quality and security. AI-assisted coding has been shown to be more beneficial for senior developers, as they possess the expertise to critically evaluate the generated code for correctness, completeness, and compliance. In contrast, junior developers may struggle to identify hallucinations, missing functionality, or incorrect logic in AI-generated code. To bridge this gap, This paper introduces a novel scoring mechanism called the SBC score, which is based on a reverse generation technique that leverages the natural language generation capabilities of LLMs. Unlike direct code analysis, our approach reconstructs system requirements from AI-generated code and compares them with the original specifications to quantify accuracy. The SBC score combines semantic similarity, BLEU, and completeness analysis, providing actionable insights to developers by highlighting missing features and hallucinations. Our code and datasets are available on GitHub

Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insights

TL;DR

This paper tackles the challenge of evaluating AI-generated code by introducing a reverse-generation framework that infers requirements from code and compares them to the original specifications using the SBC score. SBC combines semantic similarity (major weight), BLEU, and completeness to assess requirement-code alignment without relying on reference implementations. Through a dataset of 90 multi-layer requirements and four open LLMs, the study demonstrates stable performance trends across models and showcases how missing or extra elements can reveal hallucinations and gaps in generated code. The proposed approach offers actionable developer insights and can be integrated into AI code assistants to improve transparency and reliability in the software development lifecycle.

Abstract

The rise of Large Language Models (LLMs) in software engineering, particularly in code generation, has garnered significant attention. However, assessing the quality of AI-generated code remains a challenge due to the inherent complexity of programming tasks and the lack of robust evaluation metrics that align well with human judgment. Traditional token-based metrics such as BLEU and ROUGE, while commonly used in natural language processing, exhibit weak correlations with human assessments in code intelligence and verification tasks. Furthermore, these metrics are primarily research focused and are not designed for seamless integration into the software development lifecycle, limiting their practical utility for developers seeking to improve code quality and security. AI-assisted coding has been shown to be more beneficial for senior developers, as they possess the expertise to critically evaluate the generated code for correctness, completeness, and compliance. In contrast, junior developers may struggle to identify hallucinations, missing functionality, or incorrect logic in AI-generated code. To bridge this gap, This paper introduces a novel scoring mechanism called the SBC score, which is based on a reverse generation technique that leverages the natural language generation capabilities of LLMs. Unlike direct code analysis, our approach reconstructs system requirements from AI-generated code and compares them with the original specifications to quantify accuracy. The SBC score combines semantic similarity, BLEU, and completeness analysis, providing actionable insights to developers by highlighting missing features and hallucinations. Our code and datasets are available on GitHub

Paper Structure

This paper contains 17 sections, 8 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment (Yang et al., yang2023geval)
  4. ICE-Score: Instructing Large Language Models to Evaluate Code (Terry, terry2024icescore)
  5. Metamorphic Prompt Testing for LLM Code Validation (Xiaoyin et al., wang2024validating )
  6. Methodology
  7. Dataset Creation
  8. Reference Implementation
  9. Choice of Open Models
  10. SBC Score Calculation
  11. A Note on Correlation Metrics
  12. Visualization and Analysis
  13. Experiments and Results
  14. Per-LLM Evaluation
  15. Cross-LLM Comparison
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Requirements from various application layers in the dataset.
  • Figure 2: Performance evaluation graph for all LLMs over three iterations.
  • Figure 3: Generated response with SBC score for Question 86 from the Deepseek Coder model.
  • Figure 4: Consolidated score with max scores for all LLMs.
  • Figure 5: Sample output for SBC score above 0.55.
  • ...and 1 more figures