Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Functional Package Management Enable Reproducible Builds at Scale? Yes

Julien Malka, Stefano Zacchiroli, Théo Zimmermann

TL;DR

The study demonstrates that functional package management enables scalable bitwise reproducible builds in a large, real-world repository (nixpkgs), rebuilding 709,816 packages across 2017–2023 to show bitwise reproducibility rates of 69%–91% and rebuildability above 99%. Using a large Diffoscope-based dataset, the authors identify embedded build dates as a major reproducibility pitfall (~15%) and reveal that most fixes are incidental rather than intentional optimizations. The work provides a public dataset of build statuses, logs, and diffoscopes, and argues for infrastructure investments (binary caches, attestations, verification) to realize RB at scale. It also reports substantial ecosystem-level variability in reproducibility and discusses QA monitoring as a practical lever to improve sustained reproducibility across large package sets.

Abstract

Reproducible Builds (R-B) guarantee that rebuilding a software package from source leads to bitwise identical artifacts. R-B is a promising approach to increase the integrity of the software supply chain, when installing open source software built by third parties. Unfortunately, despite success stories like high build reproducibility levels in Debian packages, uncertainty remains among field experts on the scalability of R-B to very large package repositories. In this work, we perform the first large-scale study of bitwise reproducibility, in the context of the Nix functional package manager, rebuilding 709 816 packages from historical snapshots of the nixpkgs repository, the largest cross-ecosystem open source software distribution, sampled in the period 2017-2023. We obtain very high bitwise reproducibility rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%. We investigate unreproducibility causes, showing that about 15% of failures are due to embedded build dates. We release a novel dataset with all build statuses, logs, as well as full ''diffoscopes'': recursive diffs of where unreproducible build artifacts differ.

Does Functional Package Management Enable Reproducible Builds at Scale? Yes

TL;DR

The study demonstrates that functional package management enables scalable bitwise reproducible builds in a large, real-world repository (nixpkgs), rebuilding 709,816 packages across 2017–2023 to show bitwise reproducibility rates of 69%–91% and rebuildability above 99%. Using a large Diffoscope-based dataset, the authors identify embedded build dates as a major reproducibility pitfall (~15%) and reveal that most fixes are incidental rather than intentional optimizations. The work provides a public dataset of build statuses, logs, and diffoscopes, and argues for infrastructure investments (binary caches, attestations, verification) to realize RB at scale. It also reports substantial ecosystem-level variability in reproducibility and discusses QA monitoring as a practical lever to improve sustained reproducibility across large package sets.

Abstract

Reproducible Builds (R-B) guarantee that rebuilding a software package from source leads to bitwise identical artifacts. R-B is a promising approach to increase the integrity of the software supply chain, when installing open source software built by third parties. Unfortunately, despite success stories like high build reproducibility levels in Debian packages, uncertainty remains among field experts on the scalability of R-B to very large package repositories. In this work, we perform the first large-scale study of bitwise reproducibility, in the context of the Nix functional package manager, rebuilding 709 816 packages from historical snapshots of the nixpkgs repository, the largest cross-ecosystem open source software distribution, sampled in the period 2017-2023. We obtain very high bitwise reproducibility rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%. We investigate unreproducibility causes, showing that about 15% of failures are due to embedded build dates. We release a novel dataset with all build statuses, logs, as well as full ''diffoscopes'': recursive diffs of where unreproducible build artifacts differ.

Paper Structure

This paper contains 36 sections, 14 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Reproducible builds (R-B)
  4. R-B for the security of the software supply chain
  5. Techniques for build reproducibility
  6. Relaxing the bitwise reproducibility criterion
  7. Empirical studies of R-B
  8. Linux distributions and package ecosystems
  9. Background
  10. The FPM model and the Nix store
  11. Nix evaluation-build pipeline
  12. Path substitution and binary caches
  13. The nixpkgs continuous integration
  14. Testing bitwise reproducibility with Nix
  15. Diffoscope
  16. ...and 21 more sections

Figures (14)

  • Figure 1: Example Nix expression for the htop package.
  • Figure 2: Description of our build and analysis pipeline.
  • Figure 3: Evolution of the number of packages in each nixpkgs revision (as defined by release.nix), in their evaluation closure and after deduplicating ecosystem copies.
  • Figure 4: Evolution of the size of the nine most popular software ecosystems in nixpkgs, the packages whose ecosystem is undetermined (base), and the packages from other ecosystems (only packages listed in release.nix).
  • Figure 5: Number of packages introduced by every revision of the dataset and their survival in the package set over time.
  • ...and 9 more figures