Similar but Patched Code Considered Harmful -- The Impact of Similar but Patched Code on Recurring Vulnerability Detection and How to Remove Them
Zixuan Tan, Jiayuan Zhou, Xing Hu, Shengyi Pan, Kui Liu, Xin Xia
TL;DR
The paper tackles the SBP (Similar-but-Patched) phenomenon that causes clone-based vulnerability detectors to generate many false alarms. It introduces Fixed Vulnerability Filter (FVF), a language-agnostic post-processing framework that uses function-change histories and vulnerability patch logs to determine whether a detected vulnerable-looking snippet has already been patched, thereby classifying SBP cases with a containment test based on similarities like BLEU-2. Across Linux kernel and Redis, FVF reduces the false alarm rate from 76.2% to 26.6% and demonstrates the dataset's value by revealing the limited effectiveness of state-of-the-art DL-based vulnerability detectors on SBP data, while offering a real-world SBP benchmark built from 1,081 projects and 6,827 SBP functions. The work highlights the practical impact of SBP on vulnerability detection and provides a path toward more realistic evaluation and more accurate tooling, supported by a replication package.
Abstract
Identifying recurring vulnerabilities is crucial for ensuring software security. Clone-based techniques, while widely used, often generate many false alarms due to the existence of similar but patched (SBP) code, which is similar to vulnerable code but is not vulnerable due to having been patched. Although the SBP code poses a great challenge to the effectiveness of existing approaches, it has not yet been well explored. In this paper, we propose a programming language agnostic framework, Fixed Vulnerability Filter (FVF), to identify and filter such SBP instances in vulnerability detection. Different from existing studies that leverage function signatures, our approach analyzes code change histories to precisely pinpoint SBPs and consequently reduce false alarms. Evaluation under practical scenarios confirms the effectiveness and precision of our approach. Remarkably, FVF identifies and filters 65.1% of false alarms from four vulnerability detection tools (i.e., ReDeBug, VUDDY, MVP, and an elementary hash-based approach) without yielding false positives. We further apply FVF to 1,081 real-world software projects and construct a real-world SBP dataset containing 6,827 SBP functions. Due to the SBP nature, the dataset can act as a strict benchmark to test the sensitivity of the vulnerability detection approach in distinguishing real vulnerabilities and SBPs. Using this dataset, we demonstrate the ineffectiveness of four state-of-the-art deep learning-based vulnerability detection approaches. Our dataset can help developers make a more realistic evaluation of vulnerability detection approaches and also paves the way for further exploration of real-world SBP scenarios.