SyzRetrospector: A Large-Scale Retrospective Study of Syzbot

TL;DR

This paper addresses how to accurately evaluate a long-running continuous fuzzer, Syzbot, by moving beyond simple metrics like total bugs found or time-to-find. It introduces SyzRetrospector, a time-travel analysis tool that reconstructs past fuzzing environments to identify the earliest commit that reveals a bug and the factors that enable its reveal. An analysis of 559 bugs reveals that bugs spend an average of $D_1 = 331.17$ days hidden and $D_2 = 73.94$ days being revealed before finding, with an overall average delay of $\sim 405.11$ days, and finds five revealing-factor categories with varying difficulty in reducing $D_1$ and $D_2$. The study also reports practical takeaways, such as increasing compute resources and expanding syscall descriptions, to accelerate bug discovery and patching, thereby improving Syzbot’s efficiency within Linux release cadences.

Abstract

Over the past 6 years, Syzbot has fuzzed the Linux kernel day and night to report over 5570 bugs, of which 4604 have been patched [11]. While this is impressive, we have found the average time to find a bug is over 405 days. Moreover, we have found that current metrics commonly used, such as time-to-find and number of bugs found, are inaccurate in evaluating Syzbot since bugs often spend the majority of their lives hidden from the fuzzer. In this paper, we set out to better understand and quantify Syzbot's performance and improvement in finding bugs. Our tool, SyzRetrospector, takes a different approach to evaluating Syzbot by finding the earliest that Syzbot was capable of finding a bug, and why that bug was revealed. We use SyzRetrospector on a large scale to analyze 559 bugs and find that bugs are hidden for an average of 331.17 days before Syzbot is even able to find them. We further present findings on the behaviors of revealing factors, how some bugs are harder to reveal than others, the trends in delays over the past 6 years, and how bug location relates to delays. We also provide key takeaways for improving Syzbot's delays.

Figures (11)

• Figure 1: The lifetime of a bug.
• Figure 2: SyzRetrospector Workflow.
• Figure 3: SyzRetrospector determines which commit revealed the bug in question as the bug reproduces after, but not before, the revealing commit. Here that is the syscall description commit with hash $3025$. The example hashes are monotonically increasing.
• Figure 4: An example syscall description
• Figure 5: Percent of bugs revealed by each revealing factor.