Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fault Localization for Buggy Deep Learning Framework Conversions in Image Recognition

Nikolaos Louloudakis, Perry Gibson, José Cano, Ajitha Rajan

TL;DR

A novel approach towards fault localization and repair of buggy deep learning framework conversions, focusing on pre-trained image recognition models is presented, which detected a fault in a common DNN converter tool, which introduced precision errors in weights, reducing model accuracy.

Abstract

When deploying Deep Neural Networks (DNNs), developers often convert models from one deep learning framework to another (e.g., TensorFlow to PyTorch). However, this process is error-prone and can impact target model accuracy. To identify the extent of such impact, we perform and briefly present a differential analysis against three DNNs widely used for image recognition (MobileNetV2, ResNet101, and InceptionV3) converted across four well-known deep learning frameworks (PyTorch, Keras, TensorFlow (TF), and TFLite), which revealed numerous model crashes and output label discrepancies of up to 100%. To mitigate such errors, we present a novel approach towards fault localization and repair of buggy deep learning framework conversions, focusing on pre-trained image recognition models. Our technique consists of four stages of analysis: 1) conversion tools, 2) model parameters, 3) model hyperparameters, and 4) graph representation. In addition, we propose various strategies towards fault repair of the faults detected. We implement our technique on top of the Apache TVM deep learning compiler, and we test it by conducting a preliminary fault localization analysis for the conversion of InceptionV3 from TF to TFLite. Our approach detected a fault in a common DNN converter tool, which introduced precision errors in weights, reducing model accuracy. After our fault localization, we repaired the issue, reducing our conversion error to zero.

Fault Localization for Buggy Deep Learning Framework Conversions in Image Recognition

TL;DR

A novel approach towards fault localization and repair of buggy deep learning framework conversions, focusing on pre-trained image recognition models is presented, which detected a fault in a common DNN converter tool, which introduced precision errors in weights, reducing model accuracy.

Abstract

When deploying Deep Neural Networks (DNNs), developers often convert models from one deep learning framework to another (e.g., TensorFlow to PyTorch). However, this process is error-prone and can impact target model accuracy. To identify the extent of such impact, we perform and briefly present a differential analysis against three DNNs widely used for image recognition (MobileNetV2, ResNet101, and InceptionV3) converted across four well-known deep learning frameworks (PyTorch, Keras, TensorFlow (TF), and TFLite), which revealed numerous model crashes and output label discrepancies of up to 100%. To mitigate such errors, we present a novel approach towards fault localization and repair of buggy deep learning framework conversions, focusing on pre-trained image recognition models. Our technique consists of four stages of analysis: 1) conversion tools, 2) model parameters, 3) model hyperparameters, and 4) graph representation. In addition, we propose various strategies towards fault repair of the faults detected. We implement our technique on top of the Apache TVM deep learning compiler, and we test it by conducting a preliminary fault localization analysis for the conversion of InceptionV3 from TF to TFLite. Our approach detected a fault in a common DNN converter tool, which introduced precision errors in weights, reducing model accuracy. After our fault localization, we repaired the issue, reducing our conversion error to zero.
Paper Structure (13 sections, 3 figures, 1 table)

This paper contains 13 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motivation
  4. Methodology
  5. Fault Localization & Repair
  6. Conversion Tools Analysis
  7. Parameter Analysis
  8. Hyperparameter Analysis
  9. Layer Analysis
  10. Fault Repair
  11. Implementation Details
  12. Preliminary Evaluation
  13. Conclusions and Future Work

Figures (3)

  • Figure 1: Pairwise comparison of output labels between Source and converted Target models.
  • Figure 2: Fault Localization & Repair Pipeline.
  • Figure 3: Layer-wise evaluation of the differences between InceptionV3 model sourced from TensorFlow, and converted to TFLite. Parameters shows the mean difference between their weights and biases for convolutional and bias addition layers. Image 1, Image 2, and Image 3 show models' differences in activations for two inputs across Source and Target models.