An Empirical Study of Fuzz Harness Degradation
Philipp Görz, Joschua Schilling, Thorsten Holz, Marcel Böhme
TL;DR
The paper tackles the problem of fuzz harness degradation in continuous fuzzing by performing a large-scale empirical study on OSS-Fuzz across C/C++ projects. It combines quantitative analysis of coverage and bug-finding with manual case studies to identify root causes of degradation and to assess the impact of harness updates. Key findings show that overall coverage remains stable over time and that harness updates yield only modest immediate gains, though degradation occurs in a minority of cases due to build failures, code churn, or external code additions; when degradation aligns with coverage loss, bug-finding declines accordingly. The authors contribute practical detection metrics and open datasets, integrating with Fuzz Introspector and OSS-Fuzz to help maintainers proactively monitor harness health and sustain fuzzing effectiveness in evolving software ecosystems.
Abstract
The purpose of continuous fuzzing platforms is to enable fuzzing for software projects via \emph{fuzz harnesses} -- but as the projects continue to evolve, are these harnesses updated in lockstep, or do they run out of date? If these harnesses remain unmaintained, will they \emph{degrade} over time in terms of coverage achieved or number of bugs found? This is the subject of our study. We study Google's OSS-Fuzz continuous fuzzing platform containing harnesses for 510 open-source C/C++ projects, many of which are security-critical. A harness is the glue code between the fuzzer and the project, so it needs to adapt to changes in the project. It is often added by a project maintainer or as part of a, sometimes short-lived, testing effort. Our analysis shows a consistent overall fuzzer coverage percentage for projects in OSS-Fuzz and a surprising longevity of the bug-finding capability of harnesses even without explicit updates, as long as they still build. However, we also identify and manually examine individual cases of harness coverage degradation and categorize their root causes. Furthermore, we contribute to OSS-Fuzz and Fuzz Introspector to support metrics to detect harness degradation in OSS-Fuzz projects guided by this research.