Feature-Guided Analysis of Neural Networks: A Replication Study

TL;DR

This replication study evaluates Feature-Guided Analysis (FGA), a method that extracts rules from neural network internals by mapping neuron activation patterns to feature presence. The authors reimplemented FGA in TensorFlow 2.13 and extended its evaluation to MNIST and LSC benchmarks, comparing results with the original TaxiNet and YOLOv4 studies. They demonstrate that FGA achieves higher test precision on the new benchmarks, while the recall of the extracted rules is highly sensitive to neural-network choice and training, and is less affected by feature selection. The work provides a complete replication package, discusses threats to validity, and clarifies how NN selection and data composition influence rule-based explanations, underscoring FGA's practical potential for interpretable AI in diverse domains.

Abstract

Understanding why neural networks make certain decisions is pivotal for their use in safety-critical applications. Feature-Guided Analysis (FGA) extracts slices of neural networks relevant to their tasks. Existing feature-guided approaches typically monitor the activation of the neural network neurons to extract the relevant rules. Preliminary results are encouraging and demonstrate the feasibility of this solution by assessing the precision and recall of Feature-Guided Analysis on two pilot case studies. However, the applicability in industrial contexts needs additional empirical evidence. To mitigate this need, this paper assesses the applicability of FGA on a benchmark made by the MNIST and LSC datasets. We assessed the effectiveness of FGA in computing rules that explain the behavior of the neural network. Our results show that FGA has a higher precision on our benchmark than the results from the literature. We also evaluated how the selection of the neural network architecture, training, and feature selection affect the effectiveness of FGA. Our results show that the selection significantly affects the recall of FGA, while it has a negligible impact on its precision.

Figures (2)

• Figure 1: Feature-Guided Analysis: an Overview.
• Figure 2: Example of decision tree structures like the ones from FGA.