Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding Deep Learning Model in Image Recognition via Coverage Test

Wenkai Li, Xiaoqi Li, Yingjie Mao, Yishun Wang

TL;DR

An investigation was carried out on the relationships between modified decision/condition coverage and dataset size, and three potential future directions are proposed to further contribute to the security testing of DNN Models.

Abstract

Deep neural networks (DNNs) play a crucial role in the field of artificial intelligence, and their security-related testing has been a prominent research focus. By inputting test cases, the behavior of models is examined for anomalies, and coverage metrics are utilized to determine the extent of neurons covered by these test cases. With the widespread application and advancement of DNNs, different types of neural behaviors have garnered attention, leading to the emergence of various coverage metrics for neural networks. However, there is currently a lack of empirical research on these coverage metrics, specifically in analyzing the relationships and patterns between model depth, configuration information, and neural network coverage. This paper aims to investigate the relationships and patterns of four coverage metrics: primary functionality, boundary, hierarchy, and structural coverage. A series of empirical experiments were conducted, selecting LeNet, VGG, and ResNet as different DNN architectures, along with 10 models of varying depths ranging from 5 to 54 layers, to compare and study the relationships between different depths, configuration information, and various neural network coverage metrics. Additionally, an investigation was carried out on the relationships between modified decision/condition coverage and dataset size. Finally, three potential future directions are proposed to further contribute to the security testing of DNN Models.

Towards Understanding Deep Learning Model in Image Recognition via Coverage Test

TL;DR

An investigation was carried out on the relationships between modified decision/condition coverage and dataset size, and three potential future directions are proposed to further contribute to the security testing of DNN Models.

Abstract

Deep neural networks (DNNs) play a crucial role in the field of artificial intelligence, and their security-related testing has been a prominent research focus. By inputting test cases, the behavior of models is examined for anomalies, and coverage metrics are utilized to determine the extent of neurons covered by these test cases. With the widespread application and advancement of DNNs, different types of neural behaviors have garnered attention, leading to the emergence of various coverage metrics for neural networks. However, there is currently a lack of empirical research on these coverage metrics, specifically in analyzing the relationships and patterns between model depth, configuration information, and neural network coverage. This paper aims to investigate the relationships and patterns of four coverage metrics: primary functionality, boundary, hierarchy, and structural coverage. A series of empirical experiments were conducted, selecting LeNet, VGG, and ResNet as different DNN architectures, along with 10 models of varying depths ranging from 5 to 54 layers, to compare and study the relationships between different depths, configuration information, and various neural network coverage metrics. Additionally, an investigation was carried out on the relationships between modified decision/condition coverage and dataset size. Finally, three potential future directions are proposed to further contribute to the security testing of DNN Models.
Paper Structure (16 sections, 5 figures, 5 tables)

This paper contains 16 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Neuron Network Coverage
  4. Main Functional Area Coverage Indicators
  5. Layer Coverage Indicator
  6. Traditional Coverage Indicator
  7. EXPERIMENTS
  8. Experimental Setup
  9. The Neuron Coverage
  10. The K-Multisection Neuron Coverage
  11. Neuron Boundary Coverage
  12. Strong Neuron Activation Coverage
  13. Top-k Neuron Coverage
  14. Modified Condition/Decision Coverage
  15. Future Work
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Comparison of traditional code and deep neural network program flow. Above the dotted line is the traditional code flow diagram; below the dotted line is the deep neural network execution diagram. Green indicates the base cell covered by the input example.
  • Figure 2: The NC and KMNC in Different Models. Blue groups represent the coverages of LeNets, green groups for VGGs, and red groups for ResNets.
  • Figure 3: The NBC and SNAC in Different Models. Blue groups represent the coverages of LeNets, green groups for VGGs, and red groups for ResNets.
  • Figure 4: The TopKNC in Different Models. Blue groups represent the coverages of LeNets, green groups for VGGs, and red groups for ResNets.
  • Figure 5: The MC/DC Coverage.