Migrating Code At Scale With LLMs At Google

TL;DR

This work tackles the challenge of large-scale code migrations by integrating change-location discovery with an LLM-driven editing workflow and automated validation in Google's infrastructure. The proposed pipeline operates nightly, identifies both direct and indirect references to a migration target, and uses an LLM to generate code changes that are rigorously validated before submission. In a case study spanning 39 migrations over 12 months, the approach yielded 595 code changes and 93,574 edits, with the LLM responsible for about 74.45% of changes and 69.46% of edits, while developers reported substantial time savings and high satisfaction. The study demonstrates the potential of LLM-assisted automation to scale migrations, while also highlighting challenges such as context-window limits, hallucinations, and language support, suggesting directions for broader adoption and refinement.

Abstract

Developers often evolve an existing software system by making internal changes, called migration. Moving to a new framework, changing implementation to improve efficiency, and upgrading a dependency to its latest version are examples of migrations. Migration is a common and typically continuous maintenance task undertaken either manually or through tooling. Certain migrations are labor intensive and costly, developers do not find the required work rewarding, and they may take years to complete. Hence, automation is preferred for such migrations. In this paper, we discuss a large-scale, costly and traditionally manual migration project at Google, propose a novel automated algorithm that uses change location discovery and a Large Language Model (LLM) to aid developers conduct the migration, report the results of a large case study, and discuss lessons learned. Our case study on 39 distinct migrations undertaken by three developers over twelve months shows that a total of 595 code changes with 93,574 edits have been submitted, where 74.45% of the code changes and 69.46% of the edits were generated by the LLM. The developers reported high satisfaction with the automated tooling, and estimated a 50% reduction on the total time spent on the migration compared to earlier manual migrations. Our results suggest that our automated, LLM-assisted workflow can serve as a model for similar initiatives.

Figures (10)

• Figure 1: High level overview of the automated ID migration system. The system runs nightly, finds potential ID references and categorizes them. Developers make the necessary code changes using an LLM or manually (if the LLM failed to make the change), and send the changes to code owners. This process continues until no more ID references are left over to migrate.
• Figure 2: Example edit on a file in a code change and its diff in the UI of Critique critique, the web-based code review system used at Google. Deleted code is in red, added code is in green, Levenshtein levenshtein edit distance ($\Delta$) is 4.
• Figure 3: Types of code change, and how the number of edited characters by the LLM and developers are calculated.
• Figure 4: LLM$\Delta$ and Human$\Delta$ percentages for each ID.
• Figure 5: Changes in multiple languages with almost identical prompt