Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Accelerating Deep Neural Network Mutation Analysis by Neuron and Mutant Clustering

Lauren Lyons, Ali Ghanbari

TL;DR

This paper tackles the high cost of mutation analysis for deep neural networks by introducing two clustering-based acceleration techniques: neuron clustering (fewer generated mutants) and mutant clustering (fewer mutants tested). Implemented in the publicly available DeepMAACC framework, these methods leverage hierarchical clustering to group neurons or mutants and reuse testing outcomes, achieving substantial speedups. Empirical results across 8 models, 4 datasets, and 2 architectures show neuron clustering delivering about 70% speedups with larger mutation-score deviations, while mutant clustering achieves ~35% speedups with modest deviations, enabling a practical speed-accuracy trade-off. The work demonstrates tangible benefits for DNN robustness evaluation and provides a foundation for further optimization and broader applicability.

Abstract

Mutation analysis of deep neural networks (DNNs) is a promising method for effective evaluation of test data quality and model robustness, but it can be computationally expensive, especially for large models. To alleviate this, we present DEEPMAACC, a technique and a tool that speeds up DNN mutation analysis through neuron and mutant clustering. DEEPMAACC implements two methods: (1) neuron clustering to reduce the number of generated mutants and (2) mutant clustering to reduce the number of mutants to be tested by selecting representative mutants for testing. Both use hierarchical agglomerative clustering to group neurons and mutants with similar weights, with the goal of improving efficiency while maintaining mutation score. DEEPMAACC has been evaluated on 8 DNN models across 4 popular classification datasets and two DNN architectures. When compared to exhaustive, or vanilla, mutation analysis, the results provide empirical evidence that neuron clustering approach, on average, accelerates mutation analysis by 69.77%, with an average -26.84% error in mutation score. Meanwhile, mutant clustering approach, on average, accelerates mutation analysis by 35.31%, with an average 1.96% error in mutation score. Our results demonstrate that a trade-off can be made between mutation testing speed and mutation score error.

On Accelerating Deep Neural Network Mutation Analysis by Neuron and Mutant Clustering

TL;DR

This paper tackles the high cost of mutation analysis for deep neural networks by introducing two clustering-based acceleration techniques: neuron clustering (fewer generated mutants) and mutant clustering (fewer mutants tested). Implemented in the publicly available DeepMAACC framework, these methods leverage hierarchical clustering to group neurons or mutants and reuse testing outcomes, achieving substantial speedups. Empirical results across 8 models, 4 datasets, and 2 architectures show neuron clustering delivering about 70% speedups with larger mutation-score deviations, while mutant clustering achieves ~35% speedups with modest deviations, enabling a practical speed-accuracy trade-off. The work demonstrates tangible benefits for DNN robustness evaluation and provides a foundation for further optimization and broader applicability.

Abstract

Mutation analysis of deep neural networks (DNNs) is a promising method for effective evaluation of test data quality and model robustness, but it can be computationally expensive, especially for large models. To alleviate this, we present DEEPMAACC, a technique and a tool that speeds up DNN mutation analysis through neuron and mutant clustering. DEEPMAACC implements two methods: (1) neuron clustering to reduce the number of generated mutants and (2) mutant clustering to reduce the number of mutants to be tested by selecting representative mutants for testing. Both use hierarchical agglomerative clustering to group neurons and mutants with similar weights, with the goal of improving efficiency while maintaining mutation score. DEEPMAACC has been evaluated on 8 DNN models across 4 popular classification datasets and two DNN architectures. When compared to exhaustive, or vanilla, mutation analysis, the results provide empirical evidence that neuron clustering approach, on average, accelerates mutation analysis by 69.77%, with an average -26.84% error in mutation score. Meanwhile, mutant clustering approach, on average, accelerates mutation analysis by 35.31%, with an average 1.96% error in mutation score. Our results demonstrate that a trade-off can be made between mutation testing speed and mutation score error.
Paper Structure (24 sections, 3 equations, 4 figures, 1 table)

This paper contains 24 sections, 3 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Mutation Analysis of Traditional Programs
  4. Mutation Analysis of DNNs
  5. DeepMAACC Design
  6. DeepMAACC Mutators
  7. Neuron Clustering Approach
  8. Mutant Clustering Approach
  9. Experiments
  10. Datasets of DNN Models
  11. Baseline: Vanilla Approach
  12. Measures
  13. Mutation Score
  14. Mutation Testing Speedup
  15. Mutation Score Error (Loss)
  16. ...and 9 more sections

Figures (4)

  • Figure 1: An overview of approaches implemented in DeepMAACC: neuron clustering approach that creates fewer mutants by clustering the neurons and mutating neuron clusters instead of individual neurons (top row); mutant clustering approach that clusters the mutants and tests a representative from each cluster (bottom row). Chevrons represent processes, control flow is denoted by arrows, and document symbol denotes artifacts generated by the processes. Processes are marked with abbreviations for ease of reference in the text.
  • Figure 2: Number of Clustersvs.Parameter Value for the two different approaches used by DeepMAACC. Each of the four lines represent the average Number of Mutants Tested for a different DNN architecture and clustering approach. The parameter for Neuron Clustering is Neurons per Cluster, having the values of {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}. The parameter used for Mutant Clustering is ParHAC Threshold having the values of {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99}.
  • Figure 3: From the left to right: Column 1 plots the Speedup, while Column 2 shows the Mutation Score Error metric. From the top to bottom: Rows 1 and 2 reports the results of the Neuron Clustering Approach for FCNN and LeNet-5 models, respectively, and Rows 3 and 4 present the results of the Mutant Clustering Approach for FCNN and LeNet-5 models, respectively.
  • Figure 4: Box plot pairs for mutation testing Speedup for the 8 different models. In each pair, blue box-and-whisker, on the left, represents speedup for Neuron Clustering Approach, and the red box-and-whisker, on the right, represents that of Mutant Clustering Approach. Each pair of box-and-whiskers in each plot is annotated with the $p$-value obtained via Mann-Whitney U-Test. The parameter used for Neuron Clustering is Neurons per Cluster, ranging over {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, while the parameter used for Mutant Clustering is linkage threshold, ranging over {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99}. In column one, from top to bottom, the model's SpeedUp that is shown are FCNN-EMNIST, FCNN-FMNIST, FCNN-KMNIST, FCNN-MNIST, respectively. In column two, from top to bottom, the model's SpeedUp that is reported is LeNet-5-EMNIST, LeNet-5-FMNIST, LeNet-5-KMNIST, LeNet-5-MNIST, respectively.