DeepProv: Behavioral Characterization and Repair of Neural Networks via Inference Provenance Graph Analysis

TL;DR

DeepProv addresses the challenge of unpredictable DNN behavior by introducing Inference Provenance Graphs (IPGs) as a runtime, graph-based narration of inference dynamics. It combines empirical activation analytics with structural IPG representations learned by Graph Neural Networks to identify critical nodes and edges, enabling targeted, non-retraining repairs aimed at robustness, privacy, and fairness goals. Across diverse image and malware tasks, DeepProv achieves substantial robustness gains—averaging a ~55% improvement in adversarial accuracy with minimal benign score impact—by applying repair actions to select layers, often early ones. The framework is modular and adaptable to multiple repair objectives, offering practical pathways to safer, more trustworthy DNN deployments and extending to privacy auditing and fairness analysis, with open-source availability to support adoption.

Abstract

Deep neural networks (DNNs) are increasingly being deployed in high-stakes applications, from self-driving cars to biometric authentication. However, their unpredictable and unreliable behaviors in real-world settings require new approaches to characterize and ensure their reliability. This paper introduces DeepProv, a novel and customizable system designed to capture and characterize the runtime behavior of DNNs during inference by using their underlying graph structure. Inspired by system audit provenance graphs, DeepProv models the computational information flow of a DNN's inference process through Inference Provenance Graphs (IPGs). These graphs provide a detailed structural representation of the behavior of DNN, allowing both empirical and structural analysis. DeepProv uses these insights to systematically repair DNNs for specific objectives, such as improving robustness, privacy, or fairness. We instantiate DeepProv with adversarial robustness as the goal of model repair and conduct extensive case studies to evaluate its effectiveness. Our results demonstrate its effectiveness and scalability across diverse classification tasks, attack scenarios, and model complexities. DeepProv automatically identifies repair actions at the node and edge-level within IPGs, significantly enhancing the robustness of the model. In particular, applying DeepProv repair strategies to just a single layer of a DNN yields an average 55% improvement in adversarial accuracy. Moreover, DeepProv complements existing defenses, achieving substantial gains in adversarial robustness. Beyond robustness, we demonstrate the broader potential of DeepProv as an adaptable system to characterize DNN behavior in other critical areas, such as privacy auditing and fairness analysis.

Figures (9)

• Figure 1: DeepProv system overview.
• Figure 2: An illustration of IPG extraction.
• Figure 3: An illustration of DeepProv's structural characterization of IPGs.
• Figure 4: Cumulative actions analysis and filtering for the EMBER dataset.
• Figure 5: Percentage of influential nodes and edges.