Towards the Dynamics of a DNN Learning Symbolic Interactions

TL;DR

The two-phase dynamics of interactions of a deep neural network (DNN) learning interactions are mathematically proved, providing a theoretical mechanism for how the generalization power of a DNN changes during the training process.

Abstract

This study proves the two-phase dynamics of a deep neural network (DNN) learning interactions. Despite the long disappointing view of the faithfulness of post-hoc explanation of a DNN, a series of theorems have been proven in recent years to show that for a given input sample, a small set of interactions between input variables can be considered as primitive inference patterns that faithfully represent a DNN's detailed inference logic on that sample. Particularly, Zhang et al. have observed that various DNNs all learn interactions of different complexities in two distinct phases, and this two-phase dynamics well explains how a DNN changes from under-fitting to over-fitting. Therefore, in this study, we mathematically prove the two-phase dynamics of interactions, providing a theoretical mechanism for how the generalization power of a DNN changes during the training process. Experiments show that our theory well predicts the real dynamics of interactions on different DNNs trained for various tasks.

Figures (9)

• Figure 1: (a) It is proven that the DNN's inference on a certain sample is equivalent to a logical model that uses a small number of AND-OR interactions for inference. Each interaction corresponds to a non-linear (AND or OR) relationship between a set $S$ of input variables (e.g., image patches). (b) Sparsity of interactions. We show the strength $\vert I(S|\bm{x})\vert$ of all $2^n$ interactions sorted in descending order. (c) Illustration of the two-phase dynamics of a DNN learning interactions of different orders.
• Figure 2: The distribution of interaction strength $I_{\text{real}}^{(k)}$ over different orders $k$. Each row shows the change in the distribution during the training process. Experiments showed that the two-phase phenomenon widely existed on different DNNs trained on various datasets. It also verified the finding in zhang2024two that the beginning of the 2nd phase was temporally aligned with the time point when the loss gap increased. Please see Appendix \ref{['sec:apdx-more-results-two-phase']} for results on the other six DNNs trained for 3D point cloud/image/sentiment classification.
• Figure 3: Monotonic increase of $r^{(k)}$ along with $\sigma^2$ mentioned in Proposition \ref{['proposition:MS-ratio-change']}. We show the curves of $r^{(k)}$ when we set different numbers of input variables $n$ and different orders $k=1,\cdots,n-1$.
• Figure 4: Comparison between the theoretical distribution of interaction strength $I_{\text{theo}}^{(k)}$ and the real distribution of interaction strength $I_{\text{real}}^{(k)}$ in the second phase. Please see Appendix \ref{['sec:apdx-more-results-exp-verification']} for the comparison on the other six DNNs trained for 3D point cloud/image/sentiment classification.
• Figure 5: The distribution of interaction strength $I_{\text{real}}^{(k)}$ over different orders $k$. Each row shows the change of the distribution during the training process. Experiments showed that the two-phase phenomenon widely existed on different DNNs trained on various datasets.

Theorems & Definitions (17)

• Theorem 1: Sparsity property, proven by ren2024where, and discussed in Appendix \ref{['sec:apdx-condition-for-sparsity']}
• Theorem 2: Universal matching property, proven in chen2024defining and Appendix \ref{['proof:universal-matching']}
• Lemma 1: Noisy triggering function, proven in Appendix \ref{['proof:noisy-triggering']}
• Theorem 3: Proven in Appendix \ref{['proof:optimal-weights']}
• Lemma 2: Proven in Appendix \ref{['proof:J-fixed-value']}
• Theorem 4: Proven in Appendix \ref{['proof:MS-same-order-same-norm']}
• Proposition 1
• Theorem 5: Proven in Appendix \ref{['proof:no-noise']}
• proof
• Lemma 3