Fault Localization in Deep Learning-based Software: A System-level Approach

TL;DR

FL4Deep is introduced, a system-level fault localization approach considering the entire DL development pipeline to effectively localize faults across the DL-based systems.

Abstract

Over the past decade, Deep Learning (DL) has become an integral part of our daily lives. This surge in DL usage has heightened the need for developing reliable DL software systems. Given that fault localization is a critical task in reliability assessment, researchers have proposed several fault localization techniques for DL-based software, primarily focusing on faults within the DL model. While the DL model is central to DL components, there are other elements that significantly impact the performance of DL components. As a result, fault localization methods that concentrate solely on the DL model overlook a large portion of the system. To address this, we introduce FL4Deep, a system-level fault localization approach considering the entire DL development pipeline to effectively localize faults across the DL-based systems. In an evaluation using 100 faulty DL scripts, FL4Deep outperformed four previous approaches in terms of accuracy for three out of six DL-related faults, including issues related to data (84%), mismatched libraries between training and deployment (100%), and loss function (69%). Additionally, FL4Deep demonstrated superior precision and recall in fault localization for five categories of faults including three mentioned fault types in terms of accuracy, plus insufficient training iteration and activation function.

Figures (7)

• Figure 1: High-level view of a DL-based tax software system
• Figure 2: Sample KG of a DL-based system
• Figure 3: Link prediction in KG based on the type of inference (a) transductive link prediction, (b) inductive link prediction.
• Figure 4: High-level view of FL4Deep methodology
• Figure 5: Pipeline of DL-based system development process divided into three parts (adopted from amershi2019software)