Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Moving Faster and Reducing Risk: Using LLMs in Release Deployment

Rui Abreu, Vijayaraghavan Murali, Peter C Rigby, Chandra Maddila, Weiyan Sun, Jun Ge, Kaavya Chinniah, Audris Mockus, Megh Mehta, Nachiappan Nagappan

TL;DR

Models of diff risk scores (DRS) are developed to determine how likely a diff is to cause a SEV, i.e., a severe fault that impacts end-users, to predict the riskiness of an outgoing diff to cause a SEV.

Abstract

Release engineering has traditionally focused on continuously delivering features and bug fixes to users, but at a certain scale, it becomes impossible for a release engineering team to determine what should be released. At Meta's scale, the responsibility appropriately and necessarily falls back on the engineer writing and reviewing the code. To address this challenge, we developed models of diff risk scores (DRS) to determine how likely a diff is to cause a SEV, i.e., a severe fault that impacts end-users. Assuming that SEVs are only caused by diffs, a naive model could randomly gate X% of diffs from landing, which would automatically catch X% of SEVs on average. However, we aimed to build a model that can capture Y% of SEVs by gating X% of diffs, where Y >> X. By training the model on historical data on diffs that have caused SEVs in the past, we can predict the riskiness of an outgoing diff to cause a SEV. Diffs that are beyond a particular threshold of risk can then be gated. We have four types of gating: no gating (green), weekend gating (weekend), medium impact on end-users (yellow), and high impact on end-users (red). The input parameter for our models is the level of gating, and the outcome measure is the number of captured SEVs. Our research approaches include a logistic regression model, a BERT-based model, and generative LLMs. Our baseline regression model captures 18.7%, 27.9%, and 84.6% of SEVs while respectively gating the top 5% (weekend), 10% (yellow), and 50% (red) of risky diffs. The BERT-based model, StarBERT, only captures 0.61x, 0.85x, and 0.81x as many SEVs as the logistic regression for the weekend, yellow, and red gating zones, respectively. The generative LLMs, iCodeLlama-34B and iDiffLlama-13B, when risk-aligned, capture more SEVs than the logistic regression model in production: 1.40x, 1.52x, 1.05x, respectively.

Moving Faster and Reducing Risk: Using LLMs in Release Deployment

TL;DR

Models of diff risk scores (DRS) are developed to determine how likely a diff is to cause a SEV, i.e., a severe fault that impacts end-users, to predict the riskiness of an outgoing diff to cause a SEV.

Abstract

Release engineering has traditionally focused on continuously delivering features and bug fixes to users, but at a certain scale, it becomes impossible for a release engineering team to determine what should be released. At Meta's scale, the responsibility appropriately and necessarily falls back on the engineer writing and reviewing the code. To address this challenge, we developed models of diff risk scores (DRS) to determine how likely a diff is to cause a SEV, i.e., a severe fault that impacts end-users. Assuming that SEVs are only caused by diffs, a naive model could randomly gate X% of diffs from landing, which would automatically catch X% of SEVs on average. However, we aimed to build a model that can capture Y% of SEVs by gating X% of diffs, where Y >> X. By training the model on historical data on diffs that have caused SEVs in the past, we can predict the riskiness of an outgoing diff to cause a SEV. Diffs that are beyond a particular threshold of risk can then be gated. We have four types of gating: no gating (green), weekend gating (weekend), medium impact on end-users (yellow), and high impact on end-users (red). The input parameter for our models is the level of gating, and the outcome measure is the number of captured SEVs. Our research approaches include a logistic regression model, a BERT-based model, and generative LLMs. Our baseline regression model captures 18.7%, 27.9%, and 84.6% of SEVs while respectively gating the top 5% (weekend), 10% (yellow), and 50% (red) of risky diffs. The BERT-based model, StarBERT, only captures 0.61x, 0.85x, and 0.81x as many SEVs as the logistic regression for the weekend, yellow, and red gating zones, respectively. The generative LLMs, iCodeLlama-34B and iDiffLlama-13B, when risk-aligned, capture more SEVs than the logistic regression model in production: 1.40x, 1.52x, 1.05x, respectively.
Paper Structure (31 sections, 3 figures, 4 tables)

This paper contains 31 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Research Approaches
  3. Contributions and Learnings
  4. Background
  5. Software Development at Meta
  6. The Evolution of Code Freeze Practice at Meta
  7. Showing the Risk of Gated Diffs to the Developers
  8. Models
  9. Regression Model
  10. Why Large Language Models?
  11. StarBERT model
  12. Generative LLMs
  13. iCodeLlama
  14. iDiffLlama
  15. Using Generative LLMs for classification
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Showing the Risk in the Phabricator UI
  • Figure 2: Pipeline: Embeddings from foundation model. A model forward pass is run on the input diff, and hidden states from the final Transformer layer are aggregated via maxpool to form the diff's embedding $E$. An external classifier is then trained on the embeddings and labeled DRS data.
  • Figure 3: Pipeline: Risk alignment. The LLM undergoes supervised fine-tuning (SFT) on labeled DRS data, where each diff is annotated with special tokens [DRS][/DRS] that denote the model to predict risk. The LLM is trained to generate the label token 0 or 1 appended to the end of the input. During inference, the risk score is computed based on the token probabilities of the labels 0 and 1.