Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Who Said CVE? How Vulnerability Identifiers Are Mentioned by Humans, Bots, and Agents in Pull Requests

Pien Rooijendijk, Christoph Treude, Mairieli Wessel

TL;DR

This study investigates how vulnerability identifiers such as CVE, CWE, and GHSA are referenced in GitHub pull requests authored by autonomous agents, bots, and humans. Using the AIDev-pop dataset and an augmented PR collection, the authors detect IDs across PR artefacts with OSV-based patterns and classify actors with high-precision signals. They find that bots contribute the majority of ID mentions, predominantly in PR descriptions tied to automated dependency updates and audits, while humans and agents use IDs more broadly to support fixes and discussions. The results highlight distinct coordination roles for IDs across actor types and suggest tooling improvements to preserve the rationale behind ID usage and to improve traceability of security intent in automated contributions.

Abstract

Vulnerability identifiers such as CVE, CWE, and GHSA are standardised references to known software security issues, yet their use in practice is not well understood. This paper compares vulnerability ID use in GitHub pull requests authored by autonomous agents, bots, and human developers. Using the AIDev pop dataset and an augmented set of pull requests from the same repositories, we analyse who mentions vulnerability identifiers and where they appear. Bots account for around 69.1% of all mentions, usually adding few identifiers in pull request descriptions, while human and agent mentions are rarer but span more locations. Qualitative analysis shows that bots mainly reference identifiers in automated dependency updates and audits, whereas humans and agents use them to support fixes, maintenance, and discussion.

Who Said CVE? How Vulnerability Identifiers Are Mentioned by Humans, Bots, and Agents in Pull Requests

TL;DR

This study investigates how vulnerability identifiers such as CVE, CWE, and GHSA are referenced in GitHub pull requests authored by autonomous agents, bots, and humans. Using the AIDev-pop dataset and an augmented PR collection, the authors detect IDs across PR artefacts with OSV-based patterns and classify actors with high-precision signals. They find that bots contribute the majority of ID mentions, predominantly in PR descriptions tied to automated dependency updates and audits, while humans and agents use IDs more broadly to support fixes and discussions. The results highlight distinct coordination roles for IDs across actor types and suggest tooling improvements to preserve the rationale behind ID usage and to improve traceability of security intent in automated contributions.

Abstract

Vulnerability identifiers such as CVE, CWE, and GHSA are standardised references to known software security issues, yet their use in practice is not well understood. This paper compares vulnerability ID use in GitHub pull requests authored by autonomous agents, bots, and human developers. Using the AIDev pop dataset and an augmented set of pull requests from the same repositories, we analyse who mentions vulnerability identifiers and where they appear. Bots account for around 69.1% of all mentions, usually adding few identifiers in pull request descriptions, while human and agent mentions are rarer but span more locations. Qualitative analysis shows that bots mainly reference identifiers in automated dependency updates and audits, whereas humans and agents use them to support fixes, maintenance, and discussion.
Paper Structure (15 sections, 2 figures, 1 table)

This paper contains 15 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Study Design
  3. AIDev Dataset
  4. Searching Vulnerability IDs
  5. Augmenting the Dataset
  6. Identifying Bots
  7. Data Analysis
  8. Findings
  9. Automated dependency update (47 occurrences).
  10. Vulnerability fix or mitigation (12 occurrences).
  11. Dependency security audit (10 occurrences).
  12. Reference to advisory or illustrative example (1 occurrence).
  13. Threats to Validity
  14. Related work
  15. Conclusion

Figures (2)

  • Figure 1: Distribution of vulnerability ID mentions per pull request by account type (log scale)
  • Figure 2: Distribution of vulnerability ID mentions by account type and location (log scale)