Underproduction: An Approach for Measuring Risk in Open Source Software

TL;DR

The paper addresses the risk that volunteer-driven FLOSS maintenance may suffer from underproduction, where demand for reliable software outpaces available labor. It introduces a five-step conceptual framework to detect underproduction and applies it to the Debian dataset, using time-to-resolution with a Cox hazard model and popularity-based importance measured via Popcon, then quantifies misalignment with the underproduction factor $U_j$. Two experiments demonstrate widespread underproduction in Debian and validate the approach by linking higher $U_j$ to more non-maintainer uploads (NMUs). The findings highlight infrastructure risk inherent in widely-used FLOSS and suggest practical paths for targeted resource allocation and further research across repositories. The work provides a foundation for measuring, monitoring, and mitigating underproduction to improve software reliability and security in open ecosystems.

Abstract

The widespread adoption of Free/Libre and Open Source Software (FLOSS) means that the ongoing maintenance of many widely used software components relies on the collaborative effort of volunteers who set their own priorities and choose their own tasks. We argue that this has created a new form of risk that we call 'underproduction' which occurs when the supply of software engineering labor becomes out of alignment with the demand of people who rely on the software produced. We present a conceptual framework for identifying relative underproduction in software as well as a statistical method for applying our framework to a comprehensive dataset from the Debian GNU/Linux distribution that includes 21,902 source packages and the full history of 461,656 bugs. We draw on this application to present two experiments: (1) a demonstration of how our technique can be used to identify at-risk software packages in a large FLOSS repository and (2) a validation of these results using an alternate indicator of package risk. Our analysis demonstrates both the utility of our approach and reveals the existence of widespread underproduction in a range of widely-installed software components in Debian.

Figures (5)

• Figure 1: A conceptual diagram locating underproduction in relation to quality and importance.
• Figure 2: A Kaplan-Meier curve that shows the number of bugs of different severities that remain open over time.
• Figure 3: Credible intervals for $U_j$ for every package in Debian. We treat all packages whose CIs include zero as "aligned"; those whose CIs are entirely above 0 are labeled "underproduced;" those whose CIs are entirely below zero are labeled "overproduced."
• Figure 4: A heatmap of software alignment. Color intensity indicates the number of packages occupying a given ranking of quality and installation. Aligned packages appear along the lower-left to upper-right diagonal. The top heatmap includes all packages, while the bottom heatmap contains only those packages for which the 95% credible interval does not cross zero.
• Figure 5: Packages displaying the highest mean levels of underproduction. Boxplots show the mean and interquartile range of our distributions of $U_j$ and reflect uncertainty in our model of package-level quality.