FirmReBugger: A Benchmark Framework for Monolithic Firmware Fuzzers

TL;DR

FirmReBugger addresses the challenge of fairly evaluating monolithic firmware fuzzers by introducing a bug-based benchmarking framework that leverages bug oracles (Ravens) and an emulator-driven Bug Interpreter to replay fuzz seeds without altering target binaries. The framework, supported by three benchmark sets (FirmBench, FirmBenchDMA, FirmBenchX) and 313 Ravens across 61 binaries, yields ground-truth metrics such as time-to-trigger and bug-state discrimination, mitigating leaky-oracle effects and overfitting. A thorough experimental study across nine state-of-the-art fuzzers demonstrates the framework’s ability to reveal strengths and weaknesses, particularly under roadblocks like magic values, complex peripherals, DMA, and delays, while enabling community extensions through Ravens. The work enhances reproducibility and comparability in firmware fuzzing research and provides a scalable path for evolving benchmarks in step with advances in the field.

Abstract

Monolithic Firmware is widespread. Unsurprisingly, fuzz testing firmware is an active research field with new advances addressing the unique challenges in the domain. However, understanding and evaluating improvements by deriving metrics such as code coverage and unique crashes are problematic, leading to a desire for a reliable bug-based benchmark. To address the need, we design and build FirmReBugger, a holistic framework for fairly assessing monolithic firmware fuzzers with a realistic, diverse, bug-based benchmark. FirmReBugger proposes using bug oracles--C syntax expressions of bug descriptors--with an interpreter to automate analysis and accurately report on bugs discovered, discriminating between states of detected, triggered, reached and not reached. Importantly, our idea of benchmarking does not modify the target binary and simply replays fuzzing seeds to isolate the benchmark implementation from the fuzzer while providing a simple means to extend with new bug oracles. Further, analyzing fuzzing roadblocks, we created FirmBench, a set of diverse, real-world binary targets with 313 software bug oracles. Incorporating our analysis of roadblocks challenging monolithic firmware fuzzing, the bench provides for rapid evaluation of future advances. We implement FirmReBugger in a FuzzBench-for-Firmware type service and use FirmBench to evaluate 9 state-of-the art monolithic firmware fuzzers in the style of a reproducibility study, using a 10 CPU-year effort, to report our findings.

Figures (17)

• Figure 1: The internal architecture of a microcontroller, illustrating the firmware interactions with peripheral devices.
• Figure 2: Number of blocks covered by Fuzzwarescharnowski2022fuzzware on the Thermostat (introduced in Pretendergustafson2019Pretender) vs. the Thermostat (patched) binary (where the exploitable bug is removed) across five 24-hour fuzzing trials. The star denotes the mean time to trigger the exploitable bug.
• Figure 3: Illustration of two crashes with the same root cause (crash occurs in the interrupt handler). Due to the differences in interrupt timing, Crash 2 has executed an additional function (func 3) and as a result, has a different call stack when compared to Crash 1. Consequently, when using the call stack to differentiate between unique crashes; Crash 1 and Crash 2 are incorrectly distinguished as unique.
• Figure 4: A simplified overview of two bugs from the Gateway binary introduced in P$^2$IMfeng2020p2im. dataBuffer1 and pinState2 are two different vulnerable buffers defined on the heap. Due to the proximity of the buffers either buffer can overflow and corrupt the same variable---the function pointer uart_handler[1].
• Figure 5: Code snippet of buffer overflow from the Heat Press binary (introduced in P$^2$IMfeng2020p2im).