A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM
Chengjie Wang, Jingzheng Wu, Hao Lyu, Xiang Ling, Tianyue Luo, Yanjun Wu, Chen Zhao
TL;DR
The paper tackles the problem of ensuring SBOMs reliably reflect standard specifications by introducing SAP, a three-stage framework that generates, extracts, and evaluates SBOMs from a large real-world corpus. Through a two-stage study (baseline and one-year follow-up) across $55,444$ SBOMs and $3,287$ repositories using $6$ tools, it finds persistent gaps in compliance, inter-tool consistency, and field-level accuracy, with policy fields like NTIA+ often missing and inter-tool agreement averaging as low as $7.84 ext{–}12.77 ext%$. The authors attribute these gaps to unclear standard constraints and divergent tool evolutions, and demonstrate SAP's utility via ground-truth construction, reproducible experiments, and open-source artifacts. The results have practical implications for SBOM interoperability, risk management in software supply chains, and guide improvements in standard conformance and tool verification processes. Overall, the work reveals systemic, volatile challenges in current SBOM ecosystems and provides a concrete framework and data to drive improvements.
Abstract
A Software Bill of Materials (SBOM) is a machine-readable artifact that systematically organizes software information, enhancing supply chain transparency and security. To facilitate the exchange and utilization of SBOMs, organizations such as the Linux Foundation and OWASP have proposed SBOM standards. Following standards, organizations have developed tools for generating and utilizing SBOMs. However, limited research has examined the adherence of these SBOM tools to standard specifications, a gap that could lead to compliance failures and disruptions in SBOM utilization. This paper presents the first large-scale, two-stage empirical analysis of the adherence gap, using our automated evaluation framework, SAP. The evaluation, comprising a baseline evaluation and a one-year longitudinal follow-up, covers 55,444 SBOMs generated by six SBOM tools from 3,287 real-world repositories. Our analysis reveals persistent, fundamental limitations in current SBOM tools: (1) inadequate compliance support with policy requirements; (2) poor tool consistencies, including inter-tool consistency rates as low as 7.84% to 12.77% for package detection across languages, and significant longitudinal inconsistency, where tools show low consistency with their own prior versions; and (3) mediocre to poor accuracy for detailed software information, e.g., accuracy of package licenses below 20%. We analyze the root causes of these gaps and provide practical solutions. All the code, replication docker image, evaluation results are open sourced at [GitHub](https://github.com/dw763j/SAP) and [Zenodo](https://doi.org/10.5281/zenodo.14998624) for further researches.