MirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing
Shiwen Ou, Yuwei Li, Lu Yu, Chengkun Wei, Tingke Wen, Qiangpu Chen, Yu Chen, Haizhi Tang, Zulie Pan
TL;DR
The paper tackles the risk of shared bugs across DL framework APIs by introducing MirrorFuzz, an LLM-powered API fuzzing pipeline that first identifies buggy APIs from GitHub issues, then finds operation- and parameter-similar APIs across frameworks, and finally synthesizes targeted test cases to trigger analogous bugs in the target APIs. By leveraging cross-framework API similarities, MirrorFuzz achieves higher code and API coverage than state-of-the-art baselines and uncovers a substantial number of real-world bugs (315 total, 262 new), with dozens fixed and CNVD IDs assigned. Key contributions include the formalDefinitions of OS and PS API similarity, a two-stage API matching strategy with Top-k and similarity thresholds, and a few-shot CoT-based test-case synthesis using bug records from similar APIs. The results demonstrate the practical impact of exploiting shared bugs for proactive, cross-framework bug detection, highlighting implications for DL framework reliability and security.
Abstract
Deep learning (DL) frameworks serve as the backbone for a wide range of artificial intelligence applications. However, bugs within DL frameworks can cascade into critical issues in higher-level applications, jeopardizing reliability and security. While numerous techniques have been proposed to detect bugs in DL frameworks, research exploring common API patterns across frameworks and the potential risks they entail remains limited. Notably, many DL frameworks expose similar APIs with overlapping input parameters and functionalities, rendering them vulnerable to shared bugs, where a flaw in one API may extend to analogous APIs in other frameworks. To address this challenge, we propose MirrorFuzz, an automated API fuzzing solution to discover shared bugs in DL frameworks. MirrorFuzz operates in three stages: First, MirrorFuzz collects historical bug data for each API within a DL framework to identify potentially buggy APIs. Second, it matches each buggy API in a specific framework with similar APIs within and across other DL frameworks. Third, it employs large language models (LLMs) to synthesize code for the API under test, leveraging the historical bug data of similar APIs to trigger analogous bugs across APIs. We implement MirrorFuzz and evaluate it on four popular DL frameworks (TensorFlow, PyTorch, OneFlow, and Jittor). Extensive evaluation demonstrates that MirrorFuzz improves code coverage by 39.92\% and 98.20\% compared to state-of-the-art methods on TensorFlow and PyTorch, respectively. Moreover, MirrorFuzz discovers 315 bugs, 262 of which are newly found, and 80 bugs are fixed, with 52 of these bugs assigned CNVD IDs.