Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Defining and Extracting generalizable interaction primitives from DNNs

Lu Chen, Siyu Lou, Benhao Huang, Quanshi Zhang

TL;DR

A new method is developed to extract interactions that are shared by these DNNs trained for the same task, and experiments show that the extracted interactions can better reflect common knowledge shared by different DNNs.

Abstract

Faithfully summarizing the knowledge encoded by a deep neural network (DNN) into a few symbolic primitive patterns without losing much information represents a core challenge in explainable AI. To this end, Ren et al. (2024) have derived a series of theorems to prove that the inference score of a DNN can be explained as a small set of interactions between input variables. However, the lack of generalization power makes it still hard to consider such interactions as faithful primitive patterns encoded by the DNN. Therefore, given different DNNs trained for the same task, we develop a new method to extract interactions that are shared by these DNNs. Experiments show that the extracted interactions can better reflect common knowledge shared by different DNNs.

Defining and Extracting generalizable interaction primitives from DNNs

TL;DR

A new method is developed to extract interactions that are shared by these DNNs trained for the same task, and experiments show that the extracted interactions can better reflect common knowledge shared by different DNNs.

Abstract

Faithfully summarizing the knowledge encoded by a deep neural network (DNN) into a few symbolic primitive patterns without losing much information represents a core challenge in explainable AI. To this end, Ren et al. (2024) have derived a series of theorems to prove that the inference score of a DNN can be explained as a small set of interactions between input variables. However, the lack of generalization power makes it still hard to consider such interactions as faithful primitive patterns encoded by the DNN. Therefore, given different DNNs trained for the same task, we develop a new method to extract interactions that are shared by these DNNs. Experiments show that the extracted interactions can better reflect common knowledge shared by different DNNs.
Paper Structure (28 sections, 4 theorems, 10 equations, 13 figures, 3 tables)

This paper contains 28 sections, 4 theorems, 10 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Generalizable interaction primitives across DNNs
  3. Preliminaries: explaining the network output with interaction primitives
  4. Faithfulness problem with interaction-based explanations
  5. Extracting generalizable interaction primitives
  6. Only achieving sparsity is not enough
  7. Extracting generalizable interactions
  8. Experiment
  9. Conclusion, discussions, and future challenges
  10. Previous literature of using interactions to explain DNNs
  11. The conditions for universal matching of the DNN output
  12. Proof of Theorems
  13. Proof of the variance of AND and OR interactions
  14. The faithfulness of the sparsity and universal-matching.
  15. Detailed explanation of \ref{['eq:final_loss']}
  16. ...and 13 more sections

Key Result

Theorem 1

As the corollary of the proven sparsity in ren2023we, the function's output on all $2^n$ masked samples $\{\mathbf{x}_S|S\subseteq N\}$ could be universally explained by the interaction primitives in $\Omega$, s.t., $|\Omega|\ll 2^n$, i.e., $\forall S\subseteq N, v(\mathbf{x}_S)=\sum_{T\subseteq S}I

Figures (13)

  • Figure 1: Distinctive and shared interactions. When we extract AND-OR interactions from two DNNs, AND interactions $S_1\!=\!\{\textit{black},\textit{dog}\}$ and $S_2\!=\!\{\textit{black},\textit{dog},\textit{over}\}$, and OR interactions $S_3\!=\!\{\textit{black}, \textit{dog}\}$ and $S_4\!=\!\{\textit{black}, \textit{over}\}$ are shared by two DNNs, while interactions, such as the AND interaction $S_5=\{\textit{they, said}\}$, are distinctive interactions encoded by a single DNN.
  • Figure 2: Strength of AND-OR interactions $\log \vert I(S|\mathbf{x}) \vert$ over different samples in a descending order. All interactions above the dash line had much more significant effect (shown in a log space) and were considered as salient interactions.
  • Figure 3: Generalization power (measured by $s_{\text{\rm{and}}}$ and $s_{\text{\rm{or}}}$) of the extracted primitives interactions.
  • Figure 4: Overall interactions and shared interactions. The red and black bars show the overall strength of positive interactions $\textit{All}^{+, (i)}(o)$ and that of negative interactions $\textit{All}^{-, (i)}(o)$ of each $o$-th order. The orange and green bars indicate the strength of positive interactions that are shared by the other DNN $\textit{Shared}^{+, (i)}(o)$ and that of the shared negative interactions $\textit{Shared}^{-, (i)}(o)$, respectively.
  • Figure 5: Visualization of the shared and distinctive interaction primitives across different DNNs. We selected some of salient interactions from the most salient $k=50$ AND-OR interactions in each DNN. The black and gray color show the AND interactions and the OR interactions, respectively. The left and right column show the distinctive interactions extracted from the $\text{\rm BERT}_{\text{\rm BASE}}$ model and the $\text{\rm BERT}_{\text{\rm LARGE}}$ model, respectively. The middle column shows the shared interactions extracted from both models. Please see \ref{['appx:more_interactions_in_fig5']} for more interactions.
  • ...and 8 more figures

Theorems & Definitions (7)

  • Theorem 1: Universal matching theorem, proved by ren2023we
  • Theorem 2: Universal matching theorem, proof in \ref{['appdix:universal']}
  • Proposition 1
  • Definition 1: Transferability of interaction primitives
  • proof
  • Theorem 3: proved by harsanyi1963simplified
  • proof