Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mock Deep Testing: Toward Separate Development of Data and Models for Deep Learning

Ruchira Manke, Mohammad Wardat, Foutse Khomh, Hridesh Rajan

TL;DR

This paper tackles the lack of independent data- and model-level testing in deep learning by introducing mock deep testing and a practical framework, KUnit, built on Keras. It defines interfaces for data preparation and model design to decouple the DL pipeline, and demonstrates automatic mock generation and verification to enable unit testing of each stage. Empirical evaluation on 50 DL programs and a user study with 36 participants show that mocks facilitate early bug detection, faster debugging, and improved code structure, while open-source tooling allows customization. The proposed approach offers a modular testing paradigm that can reduce debugging effort and promote reliable DL application development in real-world software settings.

Abstract

While deep learning (DL) has permeated, and become an integral component of many critical software systems, today software engineering research hasn't explored how to separately test data and models that are integral for DL approaches to work effectively. The main challenge in independently testing these components arises from the tight dependency between data and models. This research explores this gap, introducing our methodology of mock deep testing for unit testing of DL applications. To enable unit testing, we introduce a design paradigm that decomposes the workflow into distinct, manageable components, minimizes sequential dependencies, and modularizes key stages of the DL. For unit testing these components, we propose modeling their dependencies using mocks. This modular approach facilitates independent development and testing of the components, ensuring comprehensive quality assurance throughout the development process. We have developed KUnit, a framework for enabling mock deep testing for the Keras library. We empirically evaluated KUnit to determine the effectiveness of mocks. Our assessment of 50 DL programs obtained from Stack Overflow and GitHub shows that mocks effectively identified 10 issues in the data preparation stage and 53 issues in the model design stage. We also conducted a user study with 36 participants using KUnit to perceive the effectiveness of our approach. Participants using KUnit successfully resolved 25 issues in the data preparation stage and 38 issues in the model design stage. Our findings highlight that mock objects provide a lightweight emulation of the dependencies for unit testing, facilitating early bug detection. Lastly, to evaluate the usability of KUnit, we conducted a post-study survey. The results reveal that KUnit is helpful to DL application developers, enabling them to independently test each component effectively in different stages.

Mock Deep Testing: Toward Separate Development of Data and Models for Deep Learning

TL;DR

This paper tackles the lack of independent data- and model-level testing in deep learning by introducing mock deep testing and a practical framework, KUnit, built on Keras. It defines interfaces for data preparation and model design to decouple the DL pipeline, and demonstrates automatic mock generation and verification to enable unit testing of each stage. Empirical evaluation on 50 DL programs and a user study with 36 participants show that mocks facilitate early bug detection, faster debugging, and improved code structure, while open-source tooling allows customization. The proposed approach offers a modular testing paradigm that can reduce debugging effort and promote reliable DL application development in real-world software settings.

Abstract

While deep learning (DL) has permeated, and become an integral component of many critical software systems, today software engineering research hasn't explored how to separately test data and models that are integral for DL approaches to work effectively. The main challenge in independently testing these components arises from the tight dependency between data and models. This research explores this gap, introducing our methodology of mock deep testing for unit testing of DL applications. To enable unit testing, we introduce a design paradigm that decomposes the workflow into distinct, manageable components, minimizes sequential dependencies, and modularizes key stages of the DL. For unit testing these components, we propose modeling their dependencies using mocks. This modular approach facilitates independent development and testing of the components, ensuring comprehensive quality assurance throughout the development process. We have developed KUnit, a framework for enabling mock deep testing for the Keras library. We empirically evaluated KUnit to determine the effectiveness of mocks. Our assessment of 50 DL programs obtained from Stack Overflow and GitHub shows that mocks effectively identified 10 issues in the data preparation stage and 53 issues in the model design stage. We also conducted a user study with 36 participants using KUnit to perceive the effectiveness of our approach. Participants using KUnit successfully resolved 25 issues in the data preparation stage and 38 issues in the model design stage. Our findings highlight that mock objects provide a lightweight emulation of the dependencies for unit testing, facilitating early bug detection. Lastly, to evaluate the usability of KUnit, we conducted a post-study survey. The results reveal that KUnit is helpful to DL application developers, enabling them to independently test each component effectively in different stages.

Paper Structure

This paper contains 32 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Approach
  4. Interface Definition
  5. Interface for Data Preparation Stage
  6. Interface for Model Design Stage
  7. Mock Object Creation and Verification
  8. Data Preparation Stage
  9. Process for Mock Model Creation
  10. Verification of Data using Mock Model
  11. Model Design Stage
  12. Process for Mock Data Creation
  13. Verification of Model using Mock Data
  14. Evaluation
  15. Experimental Setup
  16. ...and 17 more sections

Figures (5)

  • Figure 1: A buggy DL program and mocks in action.
  • Figure 2: Workflow of KUnit.
  • Figure 3: Interface definition and class description.
  • Figure 4: Mock object creation for different stages.
  • Figure 5: Survey results with participants ratings.