Improving Automated Secure Code Reviews: A Synthetic Dataset for Code Vulnerability Flaws

TL;DR

This paper proposes a method to address the underrepresentation of security-related reviews in code-review datasets by generating a synthetic dataset of vulnerability-focused reviews using large language models (LLMs) trained on security-related commits. It outlines a six-step pipeline for selecting Java projects and single-file commits, filtering vulnerability-related changes, and designing robust prompts (zero-shot, chain-of-thought, and self-reflection) to produce plausible reviews that could have prompted the commits. The study also defines a two-pronged evaluation: (RQ1) assessing the accuracy of LLM-generated reviews from commits, and (RQ2) testing whether incorporating the synthetic data improves code-review models against real, security-focused test data using BLEU-4 and manual semantic/applicability metrics. If successful, the approach promises to enrich code-review datasets with high-quality security content, enabling more reliable automated analyses and comments on security-critical code changes.

Abstract

Automation of code reviews using AI models has garnered substantial attention in the software engineering community as a strategy to reduce the cost and effort associated with traditional peer review processes. These models are typically trained on extensive datasets of real-world code reviews that address diverse software development concerns, including testing, refactoring, bug fixes, performance optimization, and maintainability improvements. However, a notable limitation of these datasets is the under representation of code vulnerabilities, critical flaws that pose significant security risks, with security-focused reviews comprising a small fraction of the data. This scarcity of vulnerability-specific data restricts the effectiveness of AI models in identifying and commenting on security-critical code. To address this issue, we propose the creation of a synthetic dataset consisting of vulnerability-focused reviews that specifically comment on security flaws. Our approach leverages Large Language Models (LLMs) to generate human-like code review comments for vulnerabilities, using insights derived from code differences and commit messages. To evaluate the usefulness of the generated synthetic dataset, we plan to use it to fine-tune three existing code review models. We anticipate that the synthetic dataset will improve the performance of the original code review models.