Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Impact of SBOM Generators on Vulnerability Assessment in Python: A Comparison and a Novel Approach

Giacomo Benedetti, Serena Cofano, Alessandro Brighente, Mauro Conti

TL;DR

The first security analysis on the vulnerability detection capabilities of tools receiving SBOMs as input is performed and a novel pip-inspired solution, PIP-sbom, is proposed, which provides improved accuracy in component identification and dependency resolution.

Abstract

The Software Supply Chain (SSC) security is a critical concern for both users and developers. Recent incidents, like the SolarWinds Orion compromise, proved the widespread impact resulting from the distribution of compromised software. The reliance on open-source components, which constitute a significant portion of modern software, further exacerbates this risk. To enhance SSC security, the Software Bill of Materials (SBOM) has been promoted as a tool to increase transparency and verifiability in software composition. However, despite its promise, SBOMs are not without limitations. Current SBOM generation tools often suffer from inaccuracies in identifying components and dependencies, leading to the creation of erroneous or incomplete representations of the SSC. Despite existing studies exposing these limitations, their impact on the vulnerability detection capabilities of security tools is still unknown. In this paper, we perform the first security analysis on the vulnerability detection capabilities of tools receiving SBOMs as input. We comprehensively evaluate SBOM generation tools by providing their outputs to vulnerability identification software. Based on our results, we identify the root causes of these tools' ineffectiveness and propose PIP-sbom, a novel pip-inspired solution that addresses their shortcomings. PIP-sbom provides improved accuracy in component identification and dependency resolution. Compared to best-performing state-of-the-art tools, PIP-sbom increases the average precision and recall by 60%, and reduces by ten times the number of false positives.

The Impact of SBOM Generators on Vulnerability Assessment in Python: A Comparison and a Novel Approach

TL;DR

The first security analysis on the vulnerability detection capabilities of tools receiving SBOMs as input is performed and a novel pip-inspired solution, PIP-sbom, is proposed, which provides improved accuracy in component identification and dependency resolution.

Abstract

The Software Supply Chain (SSC) security is a critical concern for both users and developers. Recent incidents, like the SolarWinds Orion compromise, proved the widespread impact resulting from the distribution of compromised software. The reliance on open-source components, which constitute a significant portion of modern software, further exacerbates this risk. To enhance SSC security, the Software Bill of Materials (SBOM) has been promoted as a tool to increase transparency and verifiability in software composition. However, despite its promise, SBOMs are not without limitations. Current SBOM generation tools often suffer from inaccuracies in identifying components and dependencies, leading to the creation of erroneous or incomplete representations of the SSC. Despite existing studies exposing these limitations, their impact on the vulnerability detection capabilities of security tools is still unknown. In this paper, we perform the first security analysis on the vulnerability detection capabilities of tools receiving SBOMs as input. We comprehensively evaluate SBOM generation tools by providing their outputs to vulnerability identification software. Based on our results, we identify the root causes of these tools' ineffectiveness and propose PIP-sbom, a novel pip-inspired solution that addresses their shortcomings. PIP-sbom provides improved accuracy in component identification and dependency resolution. Compared to best-performing state-of-the-art tools, PIP-sbom increases the average precision and recall by 60%, and reduces by ten times the number of false positives.
Paper Structure (38 sections, 3 equations, 6 figures, 3 tables)

This paper contains 38 sections, 3 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. SBOM Generation Process
  4. Dependencies Management in Python
  5. Vulnerabilities Scanning with SBOM
  6. Experimental Setup
  7. Projects Collection
  8. SBOM generation tools selection
  9. Security Report Ground Truth
  10. SBOMs and Security Reports Generation
  11. PIP-sbom: Our Proposed SBOM Generator
  12. Dependency Network Solving
  13. Dependency Network Graph
  14. SBOM Generation
  15. Evaluation Methodology
  16. ...and 23 more sections

Figures (6)

  • Figure 1: Example of a condensed Grype scan report for a Python sbom.
  • Figure 2: Experimental setup design. This approach provides us the necessary data to evaluate our research questions.
  • Figure 3: Design of PIP-sbom. We extend the implementation of PIP to include sbom generation in the build phase.
  • Figure 4: Jaccard Similarity Distributions. Each bar represents the percentage of SBOMs that lead to identification with a certain Jaccard index range. We include the vulnerability assessment obtained with sbom generated with PIP for comparison purposes.
  • Figure 5: Precision and Recall for the vulnerability scans conducted through sbom generated by each of the selected SBOM generation tools.
  • ...and 1 more figures