S$^2$F: Principled Hybrid Testing With Fuzzing, Symbolic Execution, and Sampling

TL;DR

This paper tackles inefficiencies in hybrid testing by addressing two core issues: the sleeping behavior of symbolic executors caused by over-pruning and the underutilization of sampling. It introduces S$^2$F, a hybrid testing tool with a coordinator that maintains a lightweight execution tree and bridges fuzzing with a tailored concolic executor that supports sampling. Guided by principles that balance difficulty, reward, and sampling cost, S$^2$F applies sampling only on hard, high-reward branches and solves others, achieving better edge coverage and more unique crashes than state-of-the-art tools across 15 real-world programs, including the discovery of previously unknown crashes. The architecture shows strong potential for practical impact by reducing symbolic-execution sleeping time and enabling principled, adaptive testing strategies that combine the strengths of all three techniques.

Abstract

Hybrid testing that integrates fuzzing, symbolic execution, and sampling has demonstrated superior testing efficiency compared to individual techniques. However, the state-of-the-art (SOTA) hybrid testing tools do not fully exploit the capabilities of symbolic execution and sampling in two key aspects. First, the SOTA hybrid testing tools employ tailored symbolic execution engines that tend to over-prune branches, leading to considerable time wasted waiting for seeds from the fuzzer and missing opportunities to discover crashes. Second, existing methods do not apply sampling to the appropriate branches and therefore cannot utilize the full capability of sampling. To address these two limitations, we propose a novel hybrid testing architecture that combines the precision of conventional symbolic execution with the scalability of tailored symbolic execution engines. Based on this architecture, we propose several principles for combining fuzzing, symbolic execution, and sampling. We implement our method in a hybrid testing tool S$^2$F. To evaluate its effectiveness, we conduct extensive experiments on 15 real-world programs. Experimental results demonstrate that S$^2$F outperforms the SOTA tool, achieving an average improvement of 6.14% in edge coverage and 32.6% in discovered crashes. Notably, our tool uncovers three previously unknown crashes in real-world programs.

Figures (10)

• Figure 1: Spectrum of hybrid testing architectures.
• Figure 2: A motivating example in libxml2
• Figure 3: Architecture of our hybrid testing platform.
• Figure 4: Comparison of three techniques.
• Figure 5: Principles of hybrid testing.