Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback

Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj, Suman Saha

TL;DR

This work addresses the unreliability of LLM-generated code by integrating a retrieval-augmented context with a multi-tool self-repair loop around a single code-generating LLM. The system uses a lightweight semantic retriever to pull security-focused repair examples from a RAG memory and leverages GCC, CodeQL, and KLEE for iterative diagnostic feedback and refinement, up to three iterations per prompt. Evaluations on 3,242 programs across two models show dramatic reductions in security defects and improved compilation and semantic correctness, with DeepSeek achieving a 96% reduction in vulnerabilities and CodeLlama reducing security defects from 58.55% to 22.19%. The results demonstrate that tool-guided, feedback-rich pipelines substantially outperform standalone generation and point to future directions in adaptive repair policies and multi-model scaling for practical secure-code pipelines.

Abstract

Large Language Models (LLMs) can generate code but often introduce security vulnerabilities, logical inconsistencies, and compilation errors. Prior work demonstrates that LLMs benefit substantially from structured feedback, static analysis, retrieval augmentation, and execution-based refinement. We propose a retrieval-augmented, multi-tool repair workflow in which a single code-generating LLM iteratively refines its outputs using compiler diagnostics, CodeQL security scanning, and KLEE symbolic execution. A lightweight embedding model is used for semantic retrieval of previously successful repairs, providing security-focused examples that guide generation. Evaluated on a combined dataset of 3,242 programs generated by DeepSeek-Coder-1.3B and CodeLlama-7B, the system demonstrates significant improvements in robustness. For DeepSeek, security vulnerabilities were reduced by 96%. For the larger CodeLlama model, the critical security defect rate was decreased from 58.55% to 22.19%, highlighting the efficacy of tool-assisted self-repair even on "stubborn" models.

Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback

TL;DR

This work addresses the unreliability of LLM-generated code by integrating a retrieval-augmented context with a multi-tool self-repair loop around a single code-generating LLM. The system uses a lightweight semantic retriever to pull security-focused repair examples from a RAG memory and leverages GCC, CodeQL, and KLEE for iterative diagnostic feedback and refinement, up to three iterations per prompt. Evaluations on 3,242 programs across two models show dramatic reductions in security defects and improved compilation and semantic correctness, with DeepSeek achieving a 96% reduction in vulnerabilities and CodeLlama reducing security defects from 58.55% to 22.19%. The results demonstrate that tool-guided, feedback-rich pipelines substantially outperform standalone generation and point to future directions in adaptive repair policies and multi-model scaling for practical secure-code pipelines.

Abstract

Large Language Models (LLMs) can generate code but often introduce security vulnerabilities, logical inconsistencies, and compilation errors. Prior work demonstrates that LLMs benefit substantially from structured feedback, static analysis, retrieval augmentation, and execution-based refinement. We propose a retrieval-augmented, multi-tool repair workflow in which a single code-generating LLM iteratively refines its outputs using compiler diagnostics, CodeQL security scanning, and KLEE symbolic execution. A lightweight embedding model is used for semantic retrieval of previously successful repairs, providing security-focused examples that guide generation. Evaluated on a combined dataset of 3,242 programs generated by DeepSeek-Coder-1.3B and CodeLlama-7B, the system demonstrates significant improvements in robustness. For DeepSeek, security vulnerabilities were reduced by 96%. For the larger CodeLlama model, the critical security defect rate was decreased from 58.55% to 22.19%, highlighting the efficacy of tool-assisted self-repair even on "stubborn" models.
Paper Structure (12 sections, 4 figures, 3 tables)

This paper contains 12 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Methodology
  4. Error Taxonomy and Dataset
  5. Results
  6. Performance on DeepSeek-Coder-1.3B
  7. Performance on CodeLlama-7B
  8. Comparative Analysis: The "Stubborn Model" Phenomenon
  9. Limitations and Future Work
  10. Reinforcement Learning for Adaptive Repair Policies.
  11. Agentic Extensions and Multi-Model Scaling.
  12. Conclusion

Figures (4)

  • Figure 1: Block diagram of the proposed multi-tool feedback system. A lightweight embedding model retrieves security-focused examples from a RAG memory to augment the prompt. A single code-generating LLM produces candidate code, which is analyzed by compiler diagnostics (GCC), symbolic execution (KLEE), and static analysis (CodeQL). Diagnostic feedback is iteratively fed back to the same LLM, and successful repairs are stored back into the RAG database.
  • Figure 2: DeepSeek error reduction across categories.
  • Figure 3: CodeLlama-7B error reduction across categories.
  • Figure 4: Comparative performance of the final system.