RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian

TL;DR

RovoDev Code Reviewer tackles the practical problem of enterprise-grade, non-finetuned code review automation by employing zero-shot context-aware prompting and two quality checks within a Bitbucket-integrated workflow. Its year-long, large-scale deployment across Atlassian repositories demonstrates tangible benefits: a code resolution rate of 38.70% for generated comments, a 30.8% reduction in median PR cycle time, and a 35.6% decrease in human-written comments, complemented by qualitative feedback highlighting accurate error detection and actionable suggestions. The study also reveals that while alignment with human-written comments is limited, the automated comments meaningfully accelerate review processes and reduce human workload, underscoring the value of targeted quality controls. The findings emphasize the practicality of privacy-preserving, context-guided code review automation in large organizations and identify design priorities for improving context enrichment and actionable feedback in future work.

Abstract

Large Language Models (LLMs)-powered code review automation has the potential to transform code review workflows. Despite the advances of LLM-powered code review comment generation approaches, several practical challenges remain for designing enterprise-grade code review automation tools. In particular, this paper aims at answering the practical question: how can we design a review-guided, context-aware, quality-checked code review comment generation without fine-tuning? In this paper, we present RovoDev Code Reviewer, an enterprise-grade LLM-based code review automation tool designed and deployed at scale within Atlassian's development ecosystem with seamless integration into Atlassian's Bitbucket. Through the offline, online, user feedback evaluations over a one-year period, we conclude that RovoDev Code Reviewer is (1) effective in generating code review comments that could lead to code resolution for 38.70% (i.e., comments that triggered code changes in the subsequent commits); and (2) offers the promise of accelerating feedback cycles (i.e., decreasing the PR cycle time by 30.8%), alleviating reviewer workload (i.e., reducing the number of human-written comments by 35.6%), and improving overall software quality (i.e., finding errors with actionable suggestions).

Figures (5)

• Figure 1: An example of a code review comment generated by RovoDev Code Reviewer.
• Figure 2: An overview of our RovoDev Code Reviewer.
• Figure 3: The results of the online evaluation stage for RovoDev Code Reviewer.
• Figure 4: An evaluation of LLM-human comment alignment, measured by %HAC (% of RovoDev-generated comments that are aligned with human-written comments, capturing both location and semantic similarity).
• Figure 5: The impact of the prompt components and comment quality check on the effectiveness of RovoDev Code Reviewer. The impact is measured by the absolute percentage difference (%$_\mathrm{control}$ - %$_\mathrm{treatment}$).