Toward Understanding Deep Learning Framework Bugs
Junjie Chen, Yihua Liang, Qingchao Shen, Jiajun Jiang, Shuochuan Li
TL;DR
This work addresses the broad quality challenge of DL frameworks by conducting the largest-scale empirical study to date on 1,000 real bugs across four major frameworks, uncovering a robust root-cause and symptom taxonomy and cross-framework commonalities. It combines a five-level architectural view with manual labeling of 13 root causes and 6 symptoms, analyzes their relationships and problematic levels, and assesses the current state of DL-framework testing techniques. The authors derive actionable guidelines for bug detection and debugging, and implement TenFuzz, a prototype testing tool that demonstrates the practicality of these guidelines by discovering unknown TensorFlow bugs in a preliminary study. The findings have practical significance for improving DL framework reliability and inform the design of more targeted, effective testing and debugging approaches across frameworks.
Abstract
DL frameworks are the basis of constructing all DL programs and models, and thus their bugs could lead to the unexpected behaviors of any DL program or model relying on them. Such a wide effect demonstrates the necessity and importance of guaranteeing DL frameworks' quality. Understanding the characteristics of DL framework bugs is a fundamental step for this quality assurance task, facilitating designing effective bug detection and debugging approaches. Hence, in this work we conduct the most large-scale study on 1,000 bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch, MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework bugs associated with 5 components decomposed from DL frameworks, as well as measuring test coverage achieved by three state-of-the-art testing techniques, we obtain 12 major findings for the comprehensive understanding of DL framework bugs and the current status of existing DL framework testing practice, and then provide a series of actionable guidelines for better DL framework bug detection and debugging. Finally, based on the guidelines, we design and implement a prototype DL-framework testing tool, called TenFuzz, which is evaluated to be effective and finds 3 unknown bugs on the latest TensorFlow framework in a preliminary study, indicating the significance of our guidelines.