Integrating Artificial Open Generative Artificial Intelligence into Software Supply Chain Security

TL;DR

The paper investigates integrating open large language models (LLMs) into software supply chain security (SCS) to address vulnerabilities and deprecated code, comparing their performance to traditional static and dynamic scanners. Using TruthfulQA as an evaluation benchmark and a balanced dataset across languages, the study analyzes cross-language effectiveness and architectural strengths. Results indicate that while LLMs can enhance monitoring and vulnerability detection, they face memory-related constraints and struggle with unseen data patterns and up-to-date information, suggesting a hybrid approach that leverages extensive security databases. Overall, the work highlights the potential of LLM-assisted SSC processes to bolster resilience, provided they operate alongside conventional tooling and continuous data updates.

Abstract

While new technologies emerge, human errors always looming. Software supply chain is increasingly complex and intertwined, the security of a service has become paramount to ensuring the integrity of products, safeguarding data privacy, and maintaining operational continuity. In this work, we conducted experiments on the promising open Large Language Models (LLMs) into two main software security challenges: source code language errors and deprecated code, with a focus on their potential to replace conventional static and dynamic security scanners that rely on predefined rules and patterns. Our findings suggest that while LLMs present some unexpected results, they also encounter significant limitations, particularly in memory complexity and the management of new and unfamiliar data patterns. Despite these challenges, the proactive application of LLMs, coupled with extensive security databases and continuous updates, holds the potential to fortify Software Supply Chain (SSC) processes against emerging threats.

Figures (2)

• Figure 1: Software Supply Chain Overview of Threats
• Figure 2: Categorical workflow of our experiments in two different categories of code quality and dependencies. After that, evaluation focuses on the performance of model and overall adaptation to software supply chain.