Docker Does Not Guarantee Reproducibility

TL;DR

This paper investigates whether Docker truly guarantees reproducible software environments. It combines a systematic literature review with a large-scale empirical study of 5298 Docker builds derived from GitHub Actions workflow runs to compare rebuilt images against their historical counterparts across multiple notions of reproducibility, including bitwise, functional, and rebuildability. The findings reveal that Docker provides no blanket guarantees: bitwise reproducibility is extremely rare, functional reproducibility is limited, and many builds fail to rebuild after time due to evolving environments. While some best practices (e.g., dependency pinning) show statistically detectable associations with improved outcomes, their effect sizes are small and insufficient as a silver bullet. The work underscores the need for careful practice, critical assessment of Docker-based reproducibility, and consideration of alternative approaches such as functional package managers to achieve stronger guarantees in scientific and software engineering contexts.

Abstract

The reproducibility of software environments is a critical concern in modern software engineering, with ramifications ranging from the effectiveness of collaboration workflows to software supply chain security and scientific reproducibility. Containerization technologies like Docker address this problem by encapsulating software environments into shareable filesystem snapshots known as images. While Docker is frequently cited in the literature as a tool that enables reproducibility in theory, the extent of its guarantees and limitations in practice remains under-explored. In this work, we address this gap through two complementary approaches. First, we conduct a systematic literature review to examine how Docker is framed in scientific discourse on reproducibility and to identify documented best practices for writing Dockerfiles enabling reproducible image building. Then, we perform a large-scale empirical study of 5298 Docker builds collected from GitHub workflows. By rebuilding these images and comparing the results with their historical counterparts, we assess the real reproducibility of Docker images and evaluate the effectiveness of the best practices identified in the literature.

Figures (6)

• Figure 1: Beliefs and expectations on Docker reproducibility extracted from the literature review.
• Figure 2: Recommendations on Dockerfiles and images extracted from the literature review.
• Figure 3: Filtering steps used to construct the workflow runs dataset used in our experiment.
• Figure 4: Number of images fully verifying each reproducibility metric, among the images for which we could retrieve the historically built version.
• Figure 5: Number of images achieving at least a given reproducibility rate on each of our four metrics.