Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively

Yao Lu, Peixin Zhang, Jingyi Wang, Lei Ma, Xiaoniu Yang, Qi Xuan

TL;DR

A novel testing approach, i.e., RedTest, is presented, which proposes a novel testing metric called Model Structural Redundancy Score (MSRS) to quantitatively measure the degree of redundancy in a deep learning model structure, and is effective in both revealing and assessing the redundancy issues in many state-of-the-art models.

Abstract

Deep learning has revolutionized computing in many real-world applications, arguably due to its remarkable performance and extreme convenience as an end-to-end solution. However, deep learning models can be costly to train and to use, especially for those large-scale models, making it necessary to optimize the original overly complicated models into smaller ones in scenarios with limited resources such as mobile applications or simply for resource saving. The key question in such model optimization is, how can we effectively identify and measure the redundancy in a deep learning model structure. While several common metrics exist in the popular model optimization techniques to measure the performance of models after optimization, they are not able to quantitatively inform the degree of remaining redundancy. To address the problem, we present a novel testing approach, i.e., RedTest, which proposes a novel testing metric called Model Structural Redundancy Score (MSRS) to quantitatively measure the degree of redundancy in a deep learning model structure. We first show that MSRS is effective in both revealing and assessing the redundancy issues in many state-of-the-art models, which urgently calls for model optimization. Then, we utilize MSRS to assist deep learning model developers in two practical application scenarios: 1) in Neural Architecture Search, we design a novel redundancy-aware algorithm to guide the search for the optimal model structure and demonstrate its effectiveness by comparing it to existing standard NAS practice; 2) in the pruning of large-scale pre-trained models, we prune the redundant layers of pre-trained models with the guidance of layer similarity to derive less redundant ones of much smaller size. Extensive experimental results demonstrate that removing such redundancy has a negligible effect on the model utility.

RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively

TL;DR

A novel testing approach, i.e., RedTest, is presented, which proposes a novel testing metric called Model Structural Redundancy Score (MSRS) to quantitatively measure the degree of redundancy in a deep learning model structure, and is effective in both revealing and assessing the redundancy issues in many state-of-the-art models.

Abstract

Deep learning has revolutionized computing in many real-world applications, arguably due to its remarkable performance and extreme convenience as an end-to-end solution. However, deep learning models can be costly to train and to use, especially for those large-scale models, making it necessary to optimize the original overly complicated models into smaller ones in scenarios with limited resources such as mobile applications or simply for resource saving. The key question in such model optimization is, how can we effectively identify and measure the redundancy in a deep learning model structure. While several common metrics exist in the popular model optimization techniques to measure the performance of models after optimization, they are not able to quantitatively inform the degree of remaining redundancy. To address the problem, we present a novel testing approach, i.e., RedTest, which proposes a novel testing metric called Model Structural Redundancy Score (MSRS) to quantitatively measure the degree of redundancy in a deep learning model structure. We first show that MSRS is effective in both revealing and assessing the redundancy issues in many state-of-the-art models, which urgently calls for model optimization. Then, we utilize MSRS to assist deep learning model developers in two practical application scenarios: 1) in Neural Architecture Search, we design a novel redundancy-aware algorithm to guide the search for the optimal model structure and demonstrate its effectiveness by comparing it to existing standard NAS practice; 2) in the pruning of large-scale pre-trained models, we prune the redundant layers of pre-trained models with the guidance of layer similarity to derive less redundant ones of much smaller size. Extensive experimental results demonstrate that removing such redundancy has a negligible effect on the model utility.

Paper Structure

This paper contains 19 sections, 14 equations, 13 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Motivation and Research Objective
  4. Motivation
  5. Application Scenarios.
  6. The Proposed RedTest
  7. The MSRS Measurement
  8. AS1: Redundancy-aware NAS
  9. AS2: Redundancy-aware Layer Pruning
  10. Experimental Evaluation
  11. Conclusion
  12. Additional Experiments on Redundancy-aware NAS
  13. How to pick expected MSRS
  14. Experimental setup
  15. Dataset Description
  16. ...and 4 more sections

Figures (13)

  • Figure 1: (a): Input image ($224 \times 224$). (b)-(e): Visualization of IRs ($28 \times 28$) in layers 5 to 8. (f): Similarity matrix. $L_i$ denotes i-th layer.
  • Figure 2: An intuitive example to explain the inadequacies of existing metrics.
  • Figure 3: Representations of hyperbolic tangent function in $\left [ 0,1 \right ]$ and scaled hyperbolic tangent functions for $\epsilon = 0.8$ and various choices of $\beta$.
  • Figure 4: Differences between existing metrics and MSRS for a family of ResNets on CIFAR10. R denotes ResNet.
  • Figure 5: Structural redundancy in widely-used backbones on various datasets. To unify the coordinate axis scale, we take the binary logarithm of the calculated MSRS.
  • ...and 8 more figures