Concurrency Testing in the Linux Kernel via eBPF

TL;DR

The paper addresses the difficulty of finding Linux kernel concurrency bugs by proposing LACE, a lightweight, in-kernel controlled concurrency testing framework built atop eBPF sched_ext. By replacing hypervisor-based control with a modular, programmable scheduler and scheduling-point instrumentation, LACE enables fine-grained interleaving exploration and a two-phase fuzzing process that combines input and schedule space exploration. Empirical results show substantial gains in code coverage and bug reproduction speed, plus discovery of eight new kernel bugs (with several confirmed by developers) and strong maintainability and extensibility characteristics. The work demonstrates that near-production kernel testing can be both effective and practical, offering a path toward more reliable, scalable kernel development and broader adoption of CCT concepts in real-world OS code.

Abstract

Concurrency is vital for our critical software to meet modern performance requirements, yet concurrency bugs are notoriously difficult to detect and reproduce. Controlled Concurrency Testing (CCT) can make bugs easier to expose by enabling control over thread interleavings and systematically exploring the interleaving space through scheduling algorithms. However, existing CCT solutions for kernel code are heavyweight, leading to significant performance, maintainability and extensibility issues. In this work, we introduce LACE, a lightweight CCT framework for kernel code empowered by eBPF. Without hypervisor modification, LACE features a custom scheduler tailored for CCT algorithms to serialize non-determistic thread execution into a controlled ordering. LACE also provides a mechanism to safely inject scheduling points into the kernel for fine-grained control. Furthermore, LACE employs a two-phase mutation strategy to integrate the scheduler with a concurrency fuzzer, allowing for automated exploration of both the input and schedule space. In our evaluation, LACE achieves 38\% more branches, 57\% overhead reduction and 11.4$\times$ speed-up in bug exposure compared to the state-of-the-art kernel concurrency fuzzers. Our qualitative analysis also demonstrates the extensibility and maintainability of LACE. Furthermore, LACE discovers eight previously unknown bugs in the Linux kernel, with six confirmed by developers.

Figures (8)

• Figure 1: A patch for a partial fix for bug in jfs filesystem highlighted in blue. The additional changes needed for a complete fix are highlighted in yellow.
• Figure 2: Overview of LACE (components of LACE in blue).
• Figure 3: Thread lifecycle within the sched_ext infrastructure. Grey ovals represent the states that a thread can be in, and blue ovals represent actions that a thread can undergo.
• Figure 4: Additional scheduling points injected by LACE instrumentation.
• Figure 5: Code coverage achieved by Syzkaller, SegFuzz and LACE (N=10).