KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution

TL;DR

The kGym platform provides a SE environment for large-scale experiments on the Linux kernel, including compiling and running kernels in parallel across several virtual machines, detecting operations and crashes, inspecting logs, and querying and patching the code base.

Abstract

Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. Unlike application-level software, a systems codebase like Linux is multilingual (low-level C/Assembly/Bash/Rust); gigantic (>20 million lines); critical (impacting billions of devices worldwide), and highly concurrent (involving complex multi-threading). To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym (a platform) and kBench (a dataset). The kGym platform provides a SE environment for large-scale experiments on the Linux kernel, including compiling and running kernels in parallel across several virtual machines, detecting operations and crashes, inspecting logs, and querying and patching the code base. We use kGym to facilitate evaluation on kBench, a crash resolution benchmark drawn from real-world Linux kernel bugs. An example bug in kBench contains crashing stack traces, a bug-reproducer file, a developer-written fix, and other associated data. To understand current performance, we conduct baseline experiments by prompting LLMs to resolve Linux kernel crashes. Our initial evaluations reveal that the best performing LLM achieves 0.72% and 5.38% in the unassisted and assisted (i.e., buggy files disclosed to the model) settings, respectively. These results highlight the need for further research to enhance model performance in SE tasks. Improving performance on kBench requires models to master new learning skills, including understanding the cause of crashes and repairing faults, writing memory-safe and hardware-aware code, and understanding concurrency. As a result, this work opens up multiple avenues of research at the intersection of machine learning and systems software.

Figures (5)

• Figure 1: kGym Pipeline. Input to kGym is a kBenchSyz bug consisting of a kernel crash and a crash reproducer file. To reproduce the bug, kGym compiles the buggy kernel version and runs the reproducer file. Next, the LLM is prompted with the kernel bug (along with the crash trace) to generate potential patch(es). Each code patch is given to kGym, which then applies the patch to the buggy kernel version, compiles the entire kernel, and subsequently executes a reproducer file to check if the bug has been successfully resolved.
• Figure 2: A sample kernel bug from Syzkaller BugLink
• Figure 3: Subsystem Distribution
• Figure 4: A sample bug patch using GPT-4 Turbo. The left figure shows a stack trace with the buggy function highlighted in red. The right compares a successfully generated patch by GPT-4 Turbo vs a human developer. The developer solution first confirms that adap->fe_adap[0].fe is not null and then uses the function pointer field ops.release to deallocate the structure safely using a custom memory deallocator. In contrast, the model uses kfree in the generated patch to deallocate the object which implicitly assumes that the object was allocated memory using kmalloc.
• Figure 5: The kGym Architecture