Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural Networks

Amin Abbasishahkoo, Mahboubeh Dadkhah, Lionel Briand, Dayi Lin

TL;DR

This paper proposes and evaluates TEASMA, a comprehensive and practical methodology designed to accurately assess the adequacy of test sets for DNNs, and suggests that it provides a reliable basis for confidently deciding whether to trust test results for DNN models.

Abstract

Successful deployment of Deep Neural Networks (DNNs) requires their validation with an adequate test set to ensure a sufficient degree of confidence in test outcomes. Although well-established test adequacy assessment techniques have been proposed for DNNs, we still need to investigate their application within a comprehensive methodology for accurately predicting the fault detection ability of test sets and thus assessing their adequacy. In this paper, we propose and evaluate TEASMA, a comprehensive and practical methodology designed to accurately assess the adequacy of test sets for DNNs. In practice, TEASMA allows engineers to decide whether they can trust high-accuracy test results and thus validate the DNN before its deployment. Based on a DNN model's training set, TEASMA provides a procedure to build accurate DNN-specific prediction models of the Fault Detection Rate (FDR) of a test set using an existing adequacy metric, thus enabling its assessment. We evaluated TEASMA with four state-of-the-art test adequacy metrics: Distance-based Surprise Coverage (DSC), Likelihood-based Surprise Coverage (LSC), Input Distribution Coverage (IDC), and Mutation Score (MS). Our extensive empirical evaluation across multiple DNN models and input sets such as ImageNet, reveals a strong linear correlation between the predicted and actual FDR values derived from MS, DSC, and IDC, with minimum R^2 values of 0.94 for MS and 0.90 for DSC and IDC. Furthermore, a low average Root Mean Square Error (RMSE) of 9% between actual and predicted FDR values across all subjects, when relying on regression analysis and MS, demonstrates the latter's superior accuracy when compared to DSC and IDC, with RMSE values of 0.17 and 0.18, respectively. Overall, these results suggest that TEASMA provides a reliable basis for confidently deciding whether to trust test results for DNN models.

TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural Networks

TL;DR

This paper proposes and evaluates TEASMA, a comprehensive and practical methodology designed to accurately assess the adequacy of test sets for DNNs, and suggests that it provides a reliable basis for confidently deciding whether to trust test results for DNN models.

Abstract

Successful deployment of Deep Neural Networks (DNNs) requires their validation with an adequate test set to ensure a sufficient degree of confidence in test outcomes. Although well-established test adequacy assessment techniques have been proposed for DNNs, we still need to investigate their application within a comprehensive methodology for accurately predicting the fault detection ability of test sets and thus assessing their adequacy. In this paper, we propose and evaluate TEASMA, a comprehensive and practical methodology designed to accurately assess the adequacy of test sets for DNNs. In practice, TEASMA allows engineers to decide whether they can trust high-accuracy test results and thus validate the DNN before its deployment. Based on a DNN model's training set, TEASMA provides a procedure to build accurate DNN-specific prediction models of the Fault Detection Rate (FDR) of a test set using an existing adequacy metric, thus enabling its assessment. We evaluated TEASMA with four state-of-the-art test adequacy metrics: Distance-based Surprise Coverage (DSC), Likelihood-based Surprise Coverage (LSC), Input Distribution Coverage (IDC), and Mutation Score (MS). Our extensive empirical evaluation across multiple DNN models and input sets such as ImageNet, reveals a strong linear correlation between the predicted and actual FDR values derived from MS, DSC, and IDC, with minimum R^2 values of 0.94 for MS and 0.90 for DSC and IDC. Furthermore, a low average Root Mean Square Error (RMSE) of 9% between actual and predicted FDR values across all subjects, when relying on regression analysis and MS, demonstrates the latter's superior accuracy when compared to DSC and IDC, with RMSE values of 0.17 and 0.18, respectively. Overall, these results suggest that TEASMA provides a reliable basis for confidently deciding whether to trust test results for DNN models.
Paper Structure (35 sections, 6 equations, 10 figures, 6 tables, 2 algorithms)

This paper contains 35 sections, 6 equations, 10 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Mutation Analysis
  4. Mutant Generation
  5. Mutation Analysis Cost
  6. Mutation Score Calculation
  7. Mutants and Faults
  8. Surprise Coverage
  9. Input Distribution Coverage
  10. Fault Estimation
  11. The TEASMA Methodology
  12. Calculating AS
  13. Estimating Faults
  14. Sampling Input Subsets
  15. Regression Analysis
  16. ...and 20 more sections

Figures (10)

  • Figure 1: Pre-training and post-training mutation operators
  • Figure 2: The process of test adequacy assessment with TEASMA
  • Figure 3: Configuring MS, SC, and IDC calculation in our experiments
  • Figure 4: Selected regression models across subjects using MS based on the training set
  • Figure 5: Linear regression line between $\widehat{FDR}$ and $ActualFDR$using MS
  • ...and 5 more figures