Enhancing Code Review through Fuzzing and Likely Invariants

TL;DR

FuzzSight addresses the gap between static code review and dynamic runtime analysis by using non-crashing fuzzing inputs to derive likely invariants and compare two program versions. The framework mines invariants at targeted breakpoints, measures distances between invariant sets with multiple metrics, and uses KDE to detect behavioral shifts that may indicate bugs. Empirical results show that likely invariant differences appear in 10%–58% of vulnerabilities and up to 80% of hard-to-find bugs, while regression bugs reach 75% detection when fuzzed; at the code-block level, FuzzSight outperforms SAST by a wide margin. This approach demonstrates the potential of integrating fuzzing-driven dynamic insights into early-stage code review to complement traditional static analysis and improve defect detection in practice.

Abstract

Many software projects employ manual code review to gatekeep defects and vulnerabilities in the code before integration. However, reviewers often work under time pressure and rely primarily on static inspection, leaving the dynamic aspects of the program unexplored. Dynamic analyses could reveal such behaviors, but they are rarely integrated into reviews. Among them, fuzzing is typically applied later to uncover crashing bugs. Yet its ability to exercise code with diverse inputs makes it promising for exposing non-crashing, but unexpected, behaviors earlier. Still, without suitable mechanisms to analyze program behaviors, the rich data produced during fuzzing remains inaccessible to reviewers, limiting its practical value in this context. We hypothesize that unexpected variations in program behaviors could signify potential bugs. The impact of code changes can be automatically captured at runtime. Representing program behavior as likely invariants, dynamic properties consistently observed at specific program points, can provide practical signals of behavioral changes. Such signals offer a way to distinguish between intended changes and unexpected behavioral shifts from code changes. We present FuzzSight, a framework that leverages likely invariants from non-crashing fuzzing inputs to highlight behavioral differences across program versions. By surfacing such differences, it provides insights into which code blocks may need closer attention. In our evaluation, FuzzSight flagged 75% of regression bugs and up to 80% of vulnerabilities uncovered by 24-hour fuzzing. It also outperformed SAST in identifying buggy code blocks, achieving ten times higher detection rates with fewer false alarms. In summary, FuzzSight demonstrates the potential and value of leveraging fuzzing and invariant analysis for early-stage code review, bridging static inspection with dynamic behavioral insights.

Figures (6)

• Figure 1: In typical software development lifecycle, code review and automated testing can operate in parallel, yet remain disconnected.
• Figure 2: Motivating example-- 1 a bug is introduced when modifying a function (red box). 2 identical inputs yield different traces and likely invariants across versions.
• Figure 3: FuzzSight Design. Execution traces from two program versions are collected using non-crashing inputs (e.g., from fuzzing tools). Likely invariants are then mined from each trace, and differences between the resulting likely invariant sets are measured to reveal behavioral changes.
• Figure 4: Mapping SAST warnings with changed code block using Abstract Syntax Tree (BugOSS: curl-8000)
• Figure 5: Interdependency between breakpoints via the program's control-flow can cause likely invariant differences in non-buggy breakpoints (BP3-BP4)