Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting "Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion": A Critical Review and Implications on DNN Coverage Testing

Jinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Shin Yoo, Paolo Tonella

TL;DR

This critique examines Neural Coverage (NLC), a DNN coverage criterion proposed at ICSE 2023, and questions its theoretical foundations and empirical validation. It argues that NLC can violate monotonicity and order-independence, misrepresents covariance structure through an $L_1$-norm, and suffers from layer-aggregation biases that erode true layer-wise insight. The authors propose improvements, including using the determinant of the covariance matrix as a more faithful scalar and reporting layer-wise coverage to preserve per-layer information, alongside rigorous, ground-truth-independent evaluation designs. Empirically, they demonstrate order-dependency, disproportionate layer influence, and counterexamples to ground-truth assumptions, underscoring the need for robust benchmarks and practical stopping criteria. Overall, the work offers guidance for designing more reliable DNN coverage metrics and evaluation protocols that better reflect real-world fault exposure and developer needs.

Abstract

We present a critical review of Neural Coverage (NLC), a state-of-the-art DNN coverage criterion by Yuan et al. at ICSE 2023. While NLC proposes to satisfy eight design requirements and demonstrates strong empirical performance, we question some of their theoretical and empirical assumptions. We observe that NLC deviates from core principles of coverage criteria, such as monotonicity and test suite order independence, and could more fully account for key properties of the covariance matrix. Additionally, we note threats to the validity of the empirical study, related to the ground truth ordering of test suites. Through our empirical validation, we substantiate our claims and propose improvements for future DNN coverage metrics. Finally, we conclude by discussing the implications of these insights.

Revisiting "Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion": A Critical Review and Implications on DNN Coverage Testing

TL;DR

This critique examines Neural Coverage (NLC), a DNN coverage criterion proposed at ICSE 2023, and questions its theoretical foundations and empirical validation. It argues that NLC can violate monotonicity and order-independence, misrepresents covariance structure through an -norm, and suffers from layer-aggregation biases that erode true layer-wise insight. The authors propose improvements, including using the determinant of the covariance matrix as a more faithful scalar and reporting layer-wise coverage to preserve per-layer information, alongside rigorous, ground-truth-independent evaluation designs. Empirically, they demonstrate order-dependency, disproportionate layer influence, and counterexamples to ground-truth assumptions, underscoring the need for robust benchmarks and practical stopping criteria. Overall, the work offers guidance for designing more reliable DNN coverage metrics and evaluation protocols that better reflect real-world fault exposure and developer needs.

Abstract

We present a critical review of Neural Coverage (NLC), a state-of-the-art DNN coverage criterion by Yuan et al. at ICSE 2023. While NLC proposes to satisfy eight design requirements and demonstrates strong empirical performance, we question some of their theoretical and empirical assumptions. We observe that NLC deviates from core principles of coverage criteria, such as monotonicity and test suite order independence, and could more fully account for key properties of the covariance matrix. Additionally, we note threats to the validity of the empirical study, related to the ground truth ordering of test suites. Through our empirical validation, we substantiate our claims and propose improvements for future DNN coverage metrics. Finally, we conclude by discussing the implications of these insights.
Paper Structure (30 sections, 7 equations, 1 figure, 3 tables)

This paper contains 30 sections, 7 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. DNN Coverage Testing
  4. NLC by Yuan et al. nlcpaper
  5. Suggested Requirements
  6. Design of DNN Coverage Criterion
  7. Critical Review of NLC
  8. Does NLC Qualify as a Coverage Criterion?
  9. Does NLC Reflect the Properties of a Covariance Matrix and Layer-Wise Information?
  10. Loss of Covariance Matrix Properties in NLC
  11. An Alternative Formulation Leveraging Intrinsic Properties of the Covariance Matrix
  12. Loss of Layer-Wise Information in NLC
  13. Critical Review of the Empirical Study
  14. Threats to Validity of Evaluation Based on Test Diversity
  15. Threats to Validity of Evaluation Based on Fault Revealing Capability
  16. ...and 15 more sections

Figures (1)

  • Figure 1: Visualisation of layer contributions in a log scale of ResNet50 on ImageNet

Theorems & Definitions (1)

  • Definition 1: Coverage Criterion