CITADEL: Context Similarity Based Deep Learning Framework Bug Finding
Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Shiwei Wang, Chao Shen
TL;DR
Citadel addresses the limited bug-type coverage and inefficiency of prior DL framework bug-finding tools by leveraging context similarity among API functions to propagate known bugs to analogous APIs. It combines static analysis, dynamic profiling, API matching, and test-case synthesis to generate executable tests for target APIs based on reproducible bug reports, evaluating each new test using the source bug as an oracle. Across PyTorch and TensorFlow, Citadel detects 151 API bugs (including 124 new, with 92 confirmed) and uncovers 13 performance bugs, achieving a bug-triggering rate of 35.40% and significantly faster detection (average 5.7 minutes per bug) than baselines. By enabling efficient coverage of diverse bug types and offering a complementary workflow to existing fuzzers, Citadel improves bug detection and repair efficiency in DL frameworks.
Abstract
With the application of deep learning technology, tools of DL framework testing are in high demand. Existing DL framework testing tools have limited coverage of bug types. For example, they lack the capability of effectively finding performance bugs, which are critical for DL models regarding performance, economics, and the environment. Moreover, existing tools are inefficient, generating hundreds of test cases with few trigger bugs. In this paper, we propose Citadel, a method that accelerates bug finding in terms of efficiency and effectiveness. We observe that many DL framework bugs are similar due to the similarity of operators and algorithms belonging to the same family. Orthogonal to existing bug-finding tools, Citadel aims to find new bugs that are similar to reported ones that have known test oracles. Citadel defines context similarity to measure the similarity of DL framework API pairs and automatically generates test cases with oracles for APIs that are similar to the problematic APIs in existing bug reports. Citadel effectively detects 58 and 66 API bugs on PyTorch and TensorFlow (excluding those rejected by developers or duplicates of prior reports), many of which, e.g., 13 performance bugs, cannot be detected by existing tools. Moreover, 35.40% of test cases generated by Citadel can trigger bugs significantly transcending the state-of-the-art method (3.90%).