SVA-ICL: Improving LLM-based Software Vulnerability Assessment via In-Context Learning and Information Fusion

TL;DR

This paper introduces SVA-ICL, a framework that leverages in-context learning and information fusion to improve LLM-based software vulnerability assessment by jointly utilizing source code and vulnerability descriptions. It employs a three-stage pipeline: (1) demonstration selection via semantic, syntactic, lexical, and textual similarities; (2) a carefully designed prompting template for ICL; and (3) vulnerability severity prediction using the DeepSeek-V2 LLM. On a large MegaVul-derived dataset with CVSS v3 labels, SVA-ICL surpasses state-of-the-art baselines in accuracy, F1-score, and MCC, with ablations showing the importance of demonstration quantity, ordering, and multi-modal fusion settings (notably 70/30 code/description and 60/40 syn/lex configurations). The results support the viability of LLM-based SVA with ICL and information fusion, suggesting practical impact for prioritizing security efforts and guiding future research in bimodal vulnerability assessment and prompt design.

Abstract

Context: Software vulnerability assessment (SVA) is critical for identifying, evaluating, and prioritizing security weaknesses in software applications. Objective: Despite the increasing application of large language models (LLMs) in various software engineering tasks, their effectiveness in SVA remains underexplored. Method: To address this gap, we introduce a novel approach SVA-ICL, which leverages in-context learning (ICL) to enhance LLM performance. Our approach involves the selection of high-quality demonstrations for ICL through information fusion, incorporating both source code and vulnerability descriptions. For source code, we consider semantic, lexical, and syntactic similarities, while for vulnerability descriptions, we focus on textual similarity. Based on the selected demonstrations, we construct context prompts and consider DeepSeek-V2 as the LLM for SVA-ICL. Results: We evaluate the effectiveness of SVA-ICL using a large-scale dataset comprising 12,071 C/C++ vulnerabilities. Experimental results demonstrate that SVA-ICL outperforms state-of-the-art SVA baselines in terms of Accuracy, F1-score, and MCC measures. Furthermore, ablation studies highlight the significance of component customization in SVA-ICL, such as the number of demonstrations, the demonstration ordering strategy, and the optimal fusion ratio of different modalities. Conclusion: Our findings suggest that leveraging ICL with information fusion can effectively improve the effectiveness of LLM-based SVA, warranting further research in this direction.

Figures (5)

• Figure 1: An example of using in-context learning for software vulnerability assessment
• Figure 2: Framework of our proposed approach SVA-ICL
• Figure 3: The prompt template designed by our proposed approach SVA-ICL.
• Figure 4: Two cases of the base severity returned by our proposed approach SVA-ICL (which only considers source code, only considers vulnerability description, and considers both with the best ratio setting)
• Figure 5: Comparison results between SVA-ICL and baselines in terms of different performance measures for different dataset splits.