The Promise and Reality of Continuous Integration Caching: An Empirical Study of Travis CI Builds

TL;DR

This study provides the first large-scale empirical examination of CI caching within Travis CI, analyzing 513,384 builds from 1,279 GitHub projects to assess adoption, maintenance, and build-time impact. Using logistic regression, regression discontinuity design, process mining, and issue analysis, it shows that only about 30% of projects use caching, with adoption tied to project maturity and awareness gaps. Maintenance is limited for many projects, and only about a third of adopters experience meaningful build-time reductions, while overhead and cache management challenges are common. The findings highlight that caching is not a universal solution and that practical benefits hinge on ongoing maintenance and thoughtful caching practices, informing both practitioners and tooling for more effective CI optimization.

Abstract

Continuous Integration (CI) provides early feedback by automatically building software, but long build durations can hinder developer productivity. CI services offer caching mechanisms to speed up builds by reusing infrequently changing artifacts, yet little is known about how caching is adopted in practice and what challenges it entails. In this paper, we conduct a large-scale empirical study of CI caching in Travis CI, analyzing 513,384 builds from 1,279 GitHub projects. We find that only 30% of projects adopt CI caching, and early adoption is strongly associated with project maturity, such as more dependencies, more commits, and longer CI lifespans. To understand why many projects do not adopt caching, we submitted pull requests enabling caching in non-adopting projects, and nearly half were accepted or merged. Developer feedback suggests that non- or late adoption mainly stems from limited awareness of CI caching support. We also examine cache maintenance and identify five common activities, performed by 24% of cache-enabled projects. Although one-third of projects see substantial build-time reductions, cache uploads occur in 97% of builds, and 33% of projects contain stale cached artifacts. Finally, our analysis of reported caching issues shows developers mainly struggle with corrupted or outdated caches or request broader caching features. Overall, CI caching does not help all projects, needs ongoing maintenance, and is more complex in practice than many developers expect.

Figures (3)

• Figure 1: Overview of our study
• Figure 2: Boxplot of the CI caching adoption delay
• Figure 3: A Fuzzy Graph of cache-changing activities