Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CI at Scale: Lean, Green, and Fast

Dhruva Juloori, Zhongpeng Lin, Matthew Williams, Eddy Shin, Sonal Mahajan

TL;DR

The paper tackles the challenge of landing changes rapidly while preserving a green mainline in Uber-scale monorepos. It extends SubmitQueue with an enhanced probabilistic model, NGBoost-based build-time predictions, BLRD, and a speculation-threshold mechanism, enabling efficient speculative execution and conflict-aware scheduling. In production across Go, iOS, and Android repos, the approach reduces CI resource usage by about 53%, CPU hours by 44%, and P95 waiting times by 37%, demonstrating substantial gains in efficiency and speed without compromising mainline integrity. The work combines system engineering with data-driven scheduling to enable scalable, cost-effective CI for large, high-velocity software ecosystems.

Abstract

Maintaining a "green" mainline branch, where all builds pass successfully, is crucial but challenging in fast-paced, large-scale software development environments, particularly with concurrent code changes in large monorepos. SubmitQueue, a system designed to address these challenges, speculatively executes builds and only lands changes with successful outcomes. However, despite its effectiveness, the system faces inefficiencies in resource utilization, leading to a high rate of premature build aborts and delays in landing smaller changes blocked by larger conflicting ones. This paper introduces enhancements to SubmitQueue, focusing on optimizing resource usage and improving build prioritization. Central to this is our innovative probabilistic model, which distinguishes between changes with shorter and longer build times to prioritize builds for more efficient scheduling. By leveraging a machine learning model to predict build times and incorporating this into the probabilistic framework, we expedite the landing of smaller changes blocked by conflicting larger time-consuming changes. Additionally, introducing a concept of speculation threshold ensures that only the most likely builds are executed, reducing unnecessary resource consumption. After implementing these enhancements across Uber's major monorepos (Go, iOS, and Android), we observed a reduction in Continuous Integration (CI) resource usage by approximately 53%, CPU usage by 44%, and P95 waiting times by 37%. These improvements highlight the enhanced efficiency of SubmitQueue in managing large-scale software changes while maintaining a green mainline.

CI at Scale: Lean, Green, and Fast

TL;DR

The paper tackles the challenge of landing changes rapidly while preserving a green mainline in Uber-scale monorepos. It extends SubmitQueue with an enhanced probabilistic model, NGBoost-based build-time predictions, BLRD, and a speculation-threshold mechanism, enabling efficient speculative execution and conflict-aware scheduling. In production across Go, iOS, and Android repos, the approach reduces CI resource usage by about 53%, CPU hours by 44%, and P95 waiting times by 37%, demonstrating substantial gains in efficiency and speed without compromising mainline integrity. The work combines system engineering with data-driven scheduling to enable scalable, cost-effective CI for large, high-velocity software ecosystems.

Abstract

Maintaining a "green" mainline branch, where all builds pass successfully, is crucial but challenging in fast-paced, large-scale software development environments, particularly with concurrent code changes in large monorepos. SubmitQueue, a system designed to address these challenges, speculatively executes builds and only lands changes with successful outcomes. However, despite its effectiveness, the system faces inefficiencies in resource utilization, leading to a high rate of premature build aborts and delays in landing smaller changes blocked by larger conflicting ones. This paper introduces enhancements to SubmitQueue, focusing on optimizing resource usage and improving build prioritization. Central to this is our innovative probabilistic model, which distinguishes between changes with shorter and longer build times to prioritize builds for more efficient scheduling. By leveraging a machine learning model to predict build times and incorporating this into the probabilistic framework, we expedite the landing of smaller changes blocked by conflicting larger time-consuming changes. Additionally, introducing a concept of speculation threshold ensures that only the most likely builds are executed, reducing unnecessary resource consumption. After implementing these enhancements across Uber's major monorepos (Go, iOS, and Android), we observed a reduction in Continuous Integration (CI) resource usage by approximately 53%, CPU usage by 44%, and P95 waiting times by 37%. These improvements highlight the enhanced efficiency of SubmitQueue in managing large-scale software changes while maintaining a green mainline.
Paper Structure (31 sections, 12 equations, 7 figures)

This paper contains 31 sections, 12 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Importance of a Stable Mainline
  4. Other Approaches and Their Limitations
  5. The Need for a Robust Change Management System
  6. System Overview
  7. Enumerator
  8. Profiler
  9. Prioritizer
  10. Selector
  11. Bypassing Large Diffs (BLRD)
  12. Probabilistic Model
  13. Enhanced Probabilistic Model
  14. Generalization
  15. Evaluating BLRD Eligibility
  16. ...and 16 more sections

Figures (7)

  • Figure 1: SubmitQueue Architecture
  • Figure 2: Monthly conflict rates across Go, iOS, and Android monorepos from January to June 2024.
  • Figure 3: Speculation tree of builds for conflicting changes $C_1$, $C_2$, and $C_3$ that arrive in the mentioned order Ananthanarayanan2019.
  • Figure 4: Monthly BLRD Trigger Rate for Go, Android, and iOS monorepos from January to June 2024.
  • Figure 5: Weekly trend of Builds-to-Changes ratio across Go, iOS, and Android monorepos.
  • ...and 2 more figures