A Practical Solution to Systematically Monitor Inconsistencies in SBOM-based Vulnerability Scanners

TL;DR

SBOM-based Vulnerability Scanning (SVS) remains fragile due to inconsistent use of component identifiers and varying SBOM quality. The authors propose SVS-TEST, a five-phase, test-driven methodology to systematically assess SVS-tools against ground-truth SBOMs in realistic edge cases. A case study across seven SVS-tools and 16 SBOMs reveals widespread silent failures and tool-specific limitations, highlighting the risk of a false sense of security. The work provides practical guidance for practitioners and researchers and shares test artifacts to enable ongoing monitoring of SVS capability and maturity.

Abstract

Software Bill of Materials (SBOM) provides new opportunities for automated vulnerability identification in software products. While the industry is adopting SBOM-based Vulnerability Scanning (SVS) to identify vulnerabilities, we increasingly observe inconsistencies and unexpected behavior, that result in false negatives and silent failures. In this work, we present the background necessary to understand the underlying complexity of SVS and introduce SVS-TEST, a method and tool to analyze the capability, maturity, and failure conditions of SVS-tools in real-world scenarios. We showcase the utility of SVS-TEST in a case study evaluating seven real-world SVS-tools using 16 precisely crafted SBOMs and their respective ground truth. Our results unveil significant differences in the reliability and error handling of SVS-tools; multiple SVS-tools silently fail on valid input SBOMs, creating a false sense of security. We conclude our work by highlighting implications for researchers and practitioners, including how organizations and developers of SVS-tools can utilize SVS-TEST to monitor SVS capability and maturity. All results and research artifacts are made publicly available and all findings were disclosed to the SVS-tool developers ahead of time.

Figures (1)

• Figure 1: svs-Tool Testing Architecture