Process-based Indicators of Vulnerability Re-Introducing Code Changes: An Exploratory Case Study

TL;DR

This paper tackles vulnerability reintroduction by shifting from file-level analysis to commit-level evaluation of security fixes, using longitudinal software engineering process metrics. Through a case study on ImageMagick, it combines 76 vulnerability-reintroducing commits with CVEs and computes bus factor, issue density, and issue spoilage to reveal socio-technical patterns around reintroductions. The main contributions include a ground-truth dataset of vulnerability reintroducing and fixing commits (extending BigVul and DiverseVul), a methodology that augments SZZ with LLM validation, and insights into how process dynamics relate to vulnerability reintroduction, such as stable bus factor, low issue density, and rising issue spoilage during reintroduction windows. The findings underscore the potential of integrating process metrics with code-level data to predict risky fixes and guide security-aware development practices in open-source projects.

Abstract

Software vulnerabilities often persist or re-emerge even after being fixed, revealing the complex interplay between code evolution and socio-technical factors. While source code metrics provide useful indicators of vulnerabilities, software engineering process metrics can uncover patterns that lead to their introduction. Yet few studies have explored whether process metrics can reveal risky development activities over time -- insights that are essential for anticipating and mitigating software vulnerabilities. This work highlights the critical role of process metrics along with code changes in understanding and mitigating vulnerability reintroduction. We move beyond file-level prediction and instead analyze security fixes at the commit level, focusing not only on whether a single fix introduces a vulnerability but also on the longer sequences of changes through which vulnerabilities evolve and re-emerge. Our approach emphasizes that reintroduction is rarely the result of one isolated action, but emerges from cumulative development activities and socio-technical conditions. To support this analysis, we conducted a case study on the ImageMagick project by correlating longitudinal process metrics such as bus factor, issue density, and issue spoilage with vulnerability reintroduction activities, encompassing 76 instances of reintroduced vulnerabilities. Our findings show that reintroductions often align with increased issue spoilage and fluctuating issue density, reflecting short-term inefficiencies in issue management and team responsiveness. These observations provide a foundation for broader studies that combine process and code metrics to predict risky fixes and strengthen software security.

Figures (6)

• Figure 1: Overview of the vulnerability reintroduction process. Contributors submit code to an open-source project which undergoes a software review process before being committed to the version control system. After the commit is made, a vulnerability in the contribution may be identified and disclosed to a vulnerability database (e.g., CVE, NVD). After which contributors submit new code that addresses the vulnerability while simultaneously reintroducing a vulnerability. This reintroduced vulnerability is then identified where it is patched out by a contributor in a potentially cyclical pattern.
• Figure 2: Our methodology to extract and compute longitudinal SEP metrics for vulnerability reintroducing and fixing commits in the ImageMagick project. We extract issues and pull requests from the GitHub issue tracker for ImageMagick to compute issue density and issue spoilage. Git commits are leveraged to compute bus factor and issue density, and to identify vulnerability fixing commits with the SZZ algorithm and Anthropic Claude 3.5 Sonnet from the ImageMagick GitHub repository, BigVul, and DiverseVul. Data is also collected from CVE and is leveraged to visualize the data.
• Figure 3: Anthropic Claude 3.5 Sonnet LLM system prompt used to evaluate commit pairs where one commit fixed a vulnerability and a subsequent commit potentially addressed a new vulnerability introduced by that fix.
• Figure 4: ImageMagick maintains a relatively healthy bus factor, with contributions from 4-7 unique maintainers every six months, facilitating distributed handling of vulnerability resolutions.
• Figure 5: ImageMagick's issue density peaked in 2022 with 28% issue density, but has since dropped to 16.4% as of writing. CVE-2016-4564 saw increasing issue density because the rate of new issues being opened exceeded the maintainers' closure rate. CVE-2017-16546 and CVE-2018-11625 both experienced density dips due to code deletion and the closing of numerous issues, respectively. CVE-2019-13299 showed increasing density driven by a rise in open issues.