Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment

Xin-Cheng Wen, Jiaxin Ye, Cuiyun Gao, Lianwei Wu, Qing Liao

TL;DR

The paper tackles the challenge of software vulnerability assessment under evolving CVSS standards and limited labeled data. It proposes EvalSVA, a multi-agent framework where several LLMs collaborate via designed communication strategies to assess commit-level SV across CVSS v3.1 base metrics. It also introduces the first multilingual CVSS-based vulnerability-commit dataset (C++, Python, Java) and demonstrates that EvalSVA outperforms single-agent baselines in accuracy and F1, while providing reasoned explanations to aid human experts. The results illustrate human-like reasoning, interpretability, and adaptability to CVSS updates, highlighting practical potential for scalable SV assessment.

Abstract

Software Vulnerability (SV) assessment is a crucial process of determining different aspects of SVs (e.g., attack vectors and scope) for developers to effectively prioritize efforts in vulnerability mitigation. It presents a challenging and laborious process due to the complexity of SVs and the scarcity of labeled data. To mitigate the above challenges, we introduce EvalSVA, a multi-agent evaluators team to autonomously deliberate and evaluate various aspects of SV assessment. Specifically, we propose a multi-agent-based framework to simulate vulnerability assessment strategies in real-world scenarios, which employs multiple Large Language Models (LLMs) into an integrated group to enhance the effectiveness of SV assessment in the limited data. We also design diverse communication strategies to autonomously discuss and assess different aspects of SV. Furthermore, we construct a multi-lingual SV assessment dataset based on the new standard of CVSS, comprising 699, 888, and 1,310 vulnerability-related commits in C++, Python, and Java, respectively. Our experimental results demonstrate that EvalSVA averagely outperforms the 44.12\% accuracy and 43.29\% F1 for SV assessment compared with the previous methods. It shows that EvalSVA offers a human-like process and generates both reason and answer for SV assessment. EvalSVA can also aid human experts in SV assessment, which provides more explanation and details for SV assessment.

EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment

TL;DR

The paper tackles the challenge of software vulnerability assessment under evolving CVSS standards and limited labeled data. It proposes EvalSVA, a multi-agent framework where several LLMs collaborate via designed communication strategies to assess commit-level SV across CVSS v3.1 base metrics. It also introduces the first multilingual CVSS-based vulnerability-commit dataset (C++, Python, Java) and demonstrates that EvalSVA outperforms single-agent baselines in accuracy and F1, while providing reasoned explanations to aid human experts. The results illustrate human-like reasoning, interpretability, and adaptability to CVSS updates, highlighting practical potential for scalable SV assessment.

Abstract

Software Vulnerability (SV) assessment is a crucial process of determining different aspects of SVs (e.g., attack vectors and scope) for developers to effectively prioritize efforts in vulnerability mitigation. It presents a challenging and laborious process due to the complexity of SVs and the scarcity of labeled data. To mitigate the above challenges, we introduce EvalSVA, a multi-agent evaluators team to autonomously deliberate and evaluate various aspects of SV assessment. Specifically, we propose a multi-agent-based framework to simulate vulnerability assessment strategies in real-world scenarios, which employs multiple Large Language Models (LLMs) into an integrated group to enhance the effectiveness of SV assessment in the limited data. We also design diverse communication strategies to autonomously discuss and assess different aspects of SV. Furthermore, we construct a multi-lingual SV assessment dataset based on the new standard of CVSS, comprising 699, 888, and 1,310 vulnerability-related commits in C++, Python, and Java, respectively. Our experimental results demonstrate that EvalSVA averagely outperforms the 44.12\% accuracy and 43.29\% F1 for SV assessment compared with the previous methods. It shows that EvalSVA offers a human-like process and generates both reason and answer for SV assessment. EvalSVA can also aid human experts in SV assessment, which provides more explanation and details for SV assessment.
Paper Structure (21 sections, 7 figures, 4 tables)

This paper contains 21 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Software Vulnerability Assessment Formulation
  4. Common vulnerability scoring system.
  5. SV assessment task formulation.
  6. Multi Agent Evaluators
  7. Multi Agents and Software Vulnerability Assessment
  8. Component
  9. Experiments
  10. Data Preparation
  11. Dataset Evaluation
  12. Baselines
  13. Evaluation Metrics
  14. EvalSVA Results
  15. Communication Strategy
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Figure (a) presents the vulnerability-related commit of CVE-2023-2954 CVE-2023-2954. The code shaded in red and green denote the vulnerability code and corresponding fixed code from commit, respectively. Figure (b) presents the three aspects and eight tasks of SV assessment. Figure (c) presents the three types of methods for SV assessment.
  • Figure 2: Communication strategy for SV assessment.
  • Figure 3: Expert number of AV.
  • Figure 4: Expert number of PR.
  • Figure 5: Round of AV.
  • ...and 2 more figures