Codexity: Secure AI-assisted Code Generation

TL;DR

The paper addresses security risks from AI-assisted code generation by proposing Codexity, a framework that couples LLMs with static analyzers to detect and repair vulnerabilities in generated code. It introduces two repair strategies—Iteration Repair, which iteratively queries powerful LLMs until vulnerability-free output is produced, and Preshot Repair, which seeds a strong LLM with vulnerability information from a local model to reduce API usage. In a real-world benchmark, Codexity achieves about a $60\%$ reduction in vulnerable outputs compared to a baseline, with tradeoffs between security and generation cost/time. The work demonstrates practical IDE integration and highlights future directions for efficiency gains and multi-language support.

Abstract

Despite the impressive performance of Large Language Models (LLMs) in software development activities, recent studies show the concern of introducing vulnerabilities into software codebase by AI programming assistants (e.g., Copilot, CodeWhisperer). In this work, we present Codexity, a security-focused code generation framework integrated with five LLMs. Codexity leverages the feedback of static analysis tools such as Infer and CppCheck to mitigate security vulnerabilities in LLM-generated programs. Our evaluation in a real-world benchmark with 751 automatically generated vulnerable subjects demonstrates Codexity can prevent 60% of the vulnerabilities being exposed to the software developer.

Figures (4)

• Figure 1: Conversational Prompt (Vulnerable Code is a placeholder for the code in Listing \ref{['lst:vul_code']})
• Figure 2: Codexity's Repair Strategies
• Figure 3: Dataset Construction
• Figure 4: Demo Screenshots