Towards Practical Defect-Focused Automated Code Review

TL;DR

This work tackles the practical problem of defect-focused automated code review by moving beyond snippet-level, text-similarity evaluation to an end-to-end pipeline integrated into a real-world DevOps environment. It introduces four components—code slicing, a multi-role LLM framework with Chain-of-Thought, a redundancy-filtering mechanism, and line-aware prompts—that collectively improve defect recall and reduce false alarms. Evaluated on industry-scale C++ fault data, the approach achieves up to a $10\times$ improvement in the Comprehensive Performance Index ($CPI$) over baselines, with a $2\times$ gain over standard LLMs, and demonstrates strong generalizability through language-agnostic design. The framework’s practical impact lies in robust defect detection during merges, precise line localization, and seamless integration with developers’ workflows, potentially extending to broader languages via AST-based tooling.

Abstract

The complexity of code reviews has driven efforts to automate review comments, but prior approaches oversimplify this task by treating it as snippet-level code-to-text generation and relying on text similarity metrics like BLEU for evaluation. These methods overlook repository context, real-world merge request evaluation, and defect detection, limiting their practicality. To address these issues, we explore the full automation pipeline within the online recommendation service of a company with nearly 400 million daily active users, analyzing industry-grade C++ codebases comprising hundreds of thousands of lines of code. We identify four key challenges: 1) capturing relevant context, 2) improving key bug inclusion (KBI), 3) reducing false alarm rates (FAR), and 4) integrating human workflows. To tackle these, we propose 1) code slicing algorithms for context extraction, 2) a multi-role LLM framework for KBI, 3) a filtering mechanism for FAR reduction, and 4) a novel prompt design for better human interaction. Our approach, validated on real-world merge requests from historical fault reports, achieves a 2x improvement over standard LLMs and a 10x gain over previous baselines. While the presented results focus on C++, the underlying framework design leverages language-agnostic principles (e.g., AST-based analysis), suggesting potential for broader applicability.

Figures (5)

• Figure 1: The code review automation pipeline integrated into the online service.
• Figure 2: An overview of our automated code review workflow.
• Figure 3: The multi-role system for automating code review.
• Figure 4: Venn diagram of recalled key bugs identified by different code slicing algorithms. The "All" setting represents all comments, while the "+Meta Reviewer" setting denotes multi-reviewer comments merged by the meta-reviewer. To analyze per-category performance, a breakdown across logic, security, and performance-related bugs is shown in Appendix \ref{['sec: Performance by error category']}.
• Figure 5: Runtime per merge request under different slicing algorithms and LLaMA3.1-405B as base models. The most time-consuming algorithm is Function, due to its inclusion of the largest extra context. However, all runtimes are within an acceptable range based on our analysis.