When Fuzzing Meets LLMs: Challenges and Opportunities

TL;DR

This paper tackles the problem that large language models (LLMs), while promising for fuzzing, suffer from hallucinations, limited context, and misinterpretation of fuzzing semantics, which can reduce bug-detection effectiveness. It provides a structured analysis identifying five challenges across driver synthesis, input generation, and bug detection, validated by a survey of recent top-tier works. To address these challenges, the authors propose three practical solutions—state-aware driver synthesis for connectors, cross-DBMS SQL transfer for input generation, and monitor-based bug detection—and demonstrate their value with preliminary DBMS fuzzing experiments showing improved correctness, diversity, and real-bug detection. The work offers concrete, corpus-dependent recommendations that can enhance the applicability of LLMs in fuzzing, particularly for complex systems like DBMSs, and has implications for improving fuzzing pipelines in broader software domains.

Abstract

Fuzzing, a widely-used technique for bug detection, has seen advancements through Large Language Models (LLMs). Despite their potential, LLMs face specific challenges in fuzzing. In this paper, we identified five major challenges of LLM-assisted fuzzing. To support our findings, we revisited the most recent papers from top-tier conferences, confirming that these challenges are widespread. As a remedy, we propose some actionable recommendations to help improve applying LLM in Fuzzing and conduct preliminary evaluations on DBMS fuzzing. The results demonstrate that our recommendations effectively address the identified challenges.

Figures (1)

• Figure 1: Fuzzing Workflow with LLM enhanced.