Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BitsAI-CR: Automated Code Review via LLM in Practice

Tao Sun, Jian Xu, Yuanpeng Li, Zhao Yan, Ge Zhang, Lintao Xie, Lu Geng, Zheng Wang, Yueyan Chen, Qin Lin, Wenbo Duan, Kaixin Sui

TL;DR

BitsAI-CR tackles enterprise-scale automated code review by combining a high-precision two-stage pipeline (RuleChecker followed by ReviewFilter) with a data flywheel that continuously updates a comprehensive taxonomy of 219 review rules. It introduces the Outdated Rate metric to gauge real-world usefulness and user adoption, enabling data-driven rule evolution. Empirical results show a peak precision of 75.0% in generated review comments and a Go Outdated Rate of 26.7%, with deployment at ByteDance serving over 12,000 weekly active users, validating scalability and practical impact. The work provides a blueprint for deploying automated code reviews in industry, balancing precision, throughput, and user acceptance through systematic feedback and continuous improvement.

Abstract

Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.

BitsAI-CR: Automated Code Review via LLM in Practice

TL;DR

BitsAI-CR tackles enterprise-scale automated code review by combining a high-precision two-stage pipeline (RuleChecker followed by ReviewFilter) with a data flywheel that continuously updates a comprehensive taxonomy of 219 review rules. It introduces the Outdated Rate metric to gauge real-world usefulness and user adoption, enabling data-driven rule evolution. Empirical results show a peak precision of 75.0% in generated review comments and a Go Outdated Rate of 26.7%, with deployment at ByteDance serving over 12,000 weekly active users, validating scalability and practical impact. The work provides a blueprint for deploying automated code reviews in industry, balancing precision, throughput, and user acceptance through systematic feedback and continuous improvement.

Abstract

Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.
Paper Structure (33 sections, 2 equations, 8 figures, 4 tables)

This paper contains 33 sections, 2 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Framework of BitsAI-CR
  5. The Taxonomy of Code Review Rules
  6. The Pipeline of Review Comment Generation
  7. Context Preparation
  8. RuleChecker
  9. ReviewFilter
  10. Comment Aggregation
  11. Evaluation Metrics
  12. Prioritizing Precision in Code Review Practice.
  13. Outdated Rate for Automated Evaluation.
  14. Data Flywheel: Continuous Evolution
  15. Mining Review Rules integrate Code Style Guide
  16. ...and 18 more sections

Figures (8)

  • Figure 1: The overview of BitsAI-CR framework for enhancing code review.
  • Figure 2: An Example of Input and Output in BitsAI-CR
  • Figure 3: A Correct but Superfluous Comment
  • Figure 4: BitsAI-CR settings interface for enabling review participation and default reviewer invitation.
  • Figure 5: The MR interface shows the "outdated" status of the initial review comments and BitsAI-CR's final approval ("LGTM" - Looks Good To Me), indicating that the developer successfully resolves issues through next commit.
  • ...and 3 more figures