Reproducible Builds and Insights from an Independent Verifier for Arch Linux

TL;DR

The paper addresses supply chain security in free and open source software by evaluating reproducible builds (R-B) and bootstrappable builds (B-B) as mechanisms to link source code to binary artifacts. It documents progress over the past decade and presents an independent Arch Linux rebuilderd verifier to test reproducibility, reporting a packaging issue in Certbot-related Arch packages and an upstream fix for fwupd to restore reproducibility. The authors demonstrate practical gains from identifying unreproducible artifacts and applying fixes, while underscoring the need for a broader network of independent rebuilders and routine source-code audits. Collectively, the work highlights the feasibility and importance of auditable software supply chains and calls for community-wide adoption and infrastructure support to realize 100% reproducibility in large repositories.

Abstract

Supply chain attacks have emerged as a prominent cybersecurity threat in recent years. Reproducible and bootstrappable builds have the potential to reduce such attacks significantly. In combination with independent, exhaustive and periodic source code audits, these measures can effectively eradicate compromises in the building process. In this paper we introduce both concepts, we analyze the achievements over the last ten years and explain the remaining challenges. We contribute to the reproducible builds effort by setting up a rebuilder and verifier instance to test the reproducibility of Arch Linux packages. Using the results from this instance, we uncover an unnoticed and security-relevant packaging issue affecting 16 packages related to Certbot, the recommended software to install TLS certificates from Let's Encrypt, making them unreproducible. Additionally, we find the root cause of unreproduciblity in the source code of fwupd, a critical software used to update device firmware on Linux devices, and submit an upstream patch to fix it.

Figures (2)

• Figure 1: The publisher of an open source software provides an executable ready to use. Most users would trust and run the executable based on the publisher's reputation. More cautious users may opt to review the source code. However, there is no guarantee that the provided executable comes directly from the published source code. R-B allows testing whether a provided artifact originates from a given source code. A set of independent verifiers can fetch the source code and rebuild the software following the exact same build instructions, as well as using the same dependencies. They publish whether they obtained the same artifacts and the user can now use this information to decide whether or not to trust the distributed executable. This diagram is a simplified version of Fig. 1 from reproducible-builds-paper.
• Figure 2: The rebuilderd daemon requires few resources and can be run on a small cloud instance (e.g., 1 vCPU and 1 GB of RAM). Our independent verifier (sync daemon and a small worker) runs on a modest refurbish PC with an Intel i7-4790K CPU and 16 GB of RAM. Workers need at least 32 GB of RAM to be able to build all the official packages. A significant speedup on large builds (e.g. linux kernel, web browsers, etc.) can be achieved with at least 64 GB of RAM and mounting the worker's directory as tmpfs. For the initial setup, two VMs with 16 vCPU, 64 GB of RAM and 200 GB of SSD storage were used as workers.