Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Fuzzing Open-Source Operating Systems

Kun Hu, Qicai Chen, Wenzhuo Zhang, Zilong Lu, Bihuan Chen, You Lu, Haowen Jiang, Bingkun Sun, Xin Peng, Wenyun Zhao

TL;DR

This paper delivers the first systematic survey of fuzzing open-source operating systems (OSF), organizing the literature around four OS layers—kernel, file system, driver, and hypervisor—and a three-part OSF workflow (input, fuzzing engine, running environment). It details seven steps within this workflow, analyzes distinctive challenges per layer (e.g., dependency inference, API polymorphism, and device-specific fuzzing stages), and summarizes bug types and monitoring approaches. The survey synthesizes trends from sixty OSF papers and four open-source tools, highlighting rising interest since 2015 and a kernel-focused emphasis driven by Syzkaller, with growing attention to non-kernel layers since 2020. It also outlines future directions, including LLM-assisted test case generation, dependency-aware fuzzing, embedded/Rust OS fuzzing, and cross-layer dependency modeling, to improve coverage, realism, and safety in OS fuzzing. The work provides a structured, actionable framework for researchers and practitioners aiming to advance vulnerability discovery in open-source OSs.

Abstract

Vulnerabilities in open-source operating systems (OSs) pose substantial security risks to software systems, making their detection crucial. While fuzzing has been an effective vulnerability detection technique in various domains, OS fuzzing (OSF) faces unique challenges due to OS complexity and multi-layered interaction, and has not been comprehensively reviewed. Therefore, this work systematically surveys the state-of-the-art OSF techniques, categorizes them based on the general fuzzing process, and investigates challenges specific to kernel, file system, driver, and hypervisor fuzzing. Finally, future research directions for OSF are discussed.

A Survey of Fuzzing Open-Source Operating Systems

TL;DR

This paper delivers the first systematic survey of fuzzing open-source operating systems (OSF), organizing the literature around four OS layers—kernel, file system, driver, and hypervisor—and a three-part OSF workflow (input, fuzzing engine, running environment). It details seven steps within this workflow, analyzes distinctive challenges per layer (e.g., dependency inference, API polymorphism, and device-specific fuzzing stages), and summarizes bug types and monitoring approaches. The survey synthesizes trends from sixty OSF papers and four open-source tools, highlighting rising interest since 2015 and a kernel-focused emphasis driven by Syzkaller, with growing attention to non-kernel layers since 2020. It also outlines future directions, including LLM-assisted test case generation, dependency-aware fuzzing, embedded/Rust OS fuzzing, and cross-layer dependency modeling, to improve coverage, realism, and safety in OS fuzzing. The work provides a structured, actionable framework for researchers and practitioners aiming to advance vulnerability discovery in open-source OSs.

Abstract

Vulnerabilities in open-source operating systems (OSs) pose substantial security risks to software systems, making their detection crucial. While fuzzing has been an effective vulnerability detection technique in various domains, OS fuzzing (OSF) faces unique challenges due to OS complexity and multi-layered interaction, and has not been comprehensively reviewed. Therefore, this work systematically surveys the state-of-the-art OSF techniques, categorizes them based on the general fuzzing process, and investigates challenges specific to kernel, file system, driver, and hypervisor fuzzing. Finally, future research directions for OSF are discussed.

Paper Structure

This paper contains 56 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Collection Strategy and Result
  3. Collection Strategy
  4. Scientific Literature Review
  5. Open-Source Tools Collection
  6. Collection Result Analysis
  7. Overview of OSF
  8. Classification Dimension
  9. OS Layers
  10. OSF Workflow
  11. General Workflow Characteristics of OSF
  12. Initial Seed Generation
  13. Pattern-Based
  14. Specification-Based
  15. Trace-Based
  16. ...and 41 more sections

Figures (3)

  • Figure 1: Our Collection Strategy
  • Figure 2: The Analysis Results of Paper Collection
  • Figure 3: Interfaces of Each OS Layer (where , , and denote the fuzzing interfaces for Kernel, Driver, File System and Hypervisor respectively; and is also the fuzzing interface for Driver and File System); and General Workflow of OSF (which has three modules, i.e., input, fuzzing engine, and running environment)