Incubation and Beyond: A Comparative Analysis of ASF Projects Sustainability Impacts on Software Quality

TL;DR

Incubation and Beyond examines how ASF project sustainability affects software quality across four trajectories: Retired, Graduated during incubation, Graduated post-incubation, and Bypassed incubation. Using Linåker's 16 sustainability metrics and eight SWQ metrics, the authors apply Bayesian Gaussian and Poisson regressions to 342 projects collected from 2009 to 2023, identifying trajectory-dependent effects where graduated projects exhibit the strongest sustainability–SWQ relationships during and after incubation. Retired projects show weaker links, while bypassed projects yield comparable patterns to graduated ones but do not consistently outperform them, underscoring that governance alone is insufficient and that internal project dynamics are critical. The findings motivate refined sustainability monitoring and governance strategies, including mentorship, knowledge transfer practices, and early-warning mechanisms to sustain both community health and software quality in FOSS ecosystems. Overall, the work demonstrates that incubation can amplify positive quality outcomes, but alternative self-regulated pathways also harbor substantial potential for sustaining high-quality software.

Abstract

Free and Open Source Software (FOSS) communities' sustainability, meaning to remain operational without signs of weakening or interruptions to its development, is fundamental for the resilience and continuity of society's digital infrastructure. Many digital services and products either leverage or entirely rely on FOSS in their software stack. FOSS sustainability is a multifaceted concept, and the impact of its decline on community products is less known. In this study, we sought to understand how the different aspects of FOSS sustainability impact software quality from a life-cycle perspective. Specifically, we investigate whether and how support and incubation of FOSS projects or bypassing incubation correlate with software quality outcomes. We selected 342 FOSS projects from the Apache Software Foundation that have either graduated, retired, or bypassed their incubator program. We used 16 sustainability metrics to examine their impact on eight software quality metrics. Using Bayesian data analysis, we found that our selected sustainability metrics exhibit distinct relationships with software quality across different project trajectories. Graduated projects showed the strongest sustainability-software quality (SWQ) relationship, both during and post-incubation. In contrast, retired projects showed weaker relationships, despite receiving similar governance support. Bypassed projects, while not outperforming graduated ones, showed comparable sustainability-SWQ relationships. While structured incubation strengthens sustainability and SWQ in graduated projects, retired projects struggle to maintain strong sustainability-SWQ relationships, indicating that additional factors internal and specific to projects influence sustainability. This effect was evident among bypassed projects; their self-reliant sustainability practices yielded stronger sustainability-SWQ compared to the retired ones.

Figures (11)

• Figure 1: H1 Comparison of Sustainability Metrics' Impact on SWQ Across Graduated and Retired Projects During Incubation. Cells labeled "Graduated" indicate that the given sustainability metric has a stronger effect on SWQ in graduated projects, whereas "Retired" signifies a stronger effect in retired projects. Cells marked ✘ indicate that there is no significant difference on the impact of the sustainability metric on SWQ in the different project trajectories. "NA" represents cases where computational limitations prevented evaluation.
• Figure 2: Posterior plots for the impact of growth (STA-4) on defect density (SWQ-1) for graduated (left) and retired projects (middle), and the posterior effect difference (right).
• Figure 3: Posterior plots for the effect difference of frequency of communication (COM-2) on code coverage (SWQ-2.1) for graduated and retired projects.
• Figure 4: Posterior plots for the effect difference of turnover (STA-9) on Very Large Function Size Count (SWQ-2.5) for graduated and retired projects.
• Figure 5: H2 Comparison of Sustainability Metrics' Impact on SWQ Across Graduated Projects During Incubation and Post Incubation. Cells labeled "Incubation" indicate that the given sustainability metric has a stronger effect on SWQ during incubation, whereas "Post incubation" signifies a stronger effect post incubation (or after graduation). Cells marked ✘ indicate that there is no significant difference on the impact of the sustainability metric on SWQ in the compared periods. "NA" represents cases where computational limitations prevented evaluation.