Deep Neural Networks via Complex Network Theory: a Perspective

TL;DR

This work reframes Deep Neural Networks as CNT-based graphs to interpret training dynamics beyond topology. It extends CNT metrics with data-dependent measures and derives formulations for FC, AE, CNN, and RNN architectures, enabling cross-architecture comparisons using metrics such as Link Weights, Nodes Strength, Layers Fluctuation, Neurons Strength, and Neurons Activation. The authors demonstrate that these metrics reveal distinct patterns across architectures, activations, and depths, providing a physics-inspired lens on learning dynamics. Experiments on MNIST and CIFAR-10 across 30 networks per architecture show architecture- and depth-specific CNT signatures and highlight the potential for CNT-guided interpretability and network design.

Abstract

Deep Neural Networks (DNNs) can be represented as graphs whose links and vertices iteratively process data and solve tasks sub-optimally. Complex Network Theory (CNT), merging statistical physics with graph theory, provides a method for interpreting neural networks by analysing their weights and neuron structures. However, classic works adapt CNT metrics that only permit a topological analysis as they do not account for the effect of the input data. In addition, CNT metrics have been applied to a limited range of architectures, mainly including Fully Connected neural networks. In this work, we extend the existing CNT metrics with measures that sample from the DNNs' training distribution, shifting from a purely topological analysis to one that connects with the interpretability of deep learning. For the novel metrics, in addition to the existing ones, we provide a mathematical formalisation for Fully Connected, AutoEncoder, Convolutional and Recurrent neural networks, of which we vary the activation functions and the number of hidden layers. We show that these metrics differentiate DNNs based on the architecture, the number of hidden layers, and the activation function. Our contribution provides a method rooted in physics for interpreting DNNs that offers insights beyond the traditional input-output relationship and the CNT topological analysis.

Figures (8)

• Figure 1: For CNNs, CNT metrics are computed by isolating each input patch and the kernel responsible for a dot-product in a layer (left), while for RNNs, metrics can be computed by unfolding each input feature through time (right).
• Figure 2: Analysis of CNT metrics across the second and third layers (hidden and output layers) for three-layer depth FCs, CNNs, RNNs and AEs on the MNIST dataset. Each column corresponds to an architecture, and the figures illustrate the distribution functions computed on a pool of $30$ neural networks trained on the task.
• Figure 3: Neurons Strength metric for $30$ RNNs on MNIST (top) and CIFAR10 (bottom) classification tasks, distinguishing between networks that have been trained and those that remain untrained. The left side of the figure quantifies the Neurons Strength, while the right side visualises a global heatmap of neurons that are most elicited by MNIST/CIFAR10 inputs.
• Figure 4: Analysis of CNT metrics for three-layer depth FCs, CNNs, RNNs and AEs on the CIFAR10 dataset and different activation functions (linear, ReLU and sigmoid). Each column corresponds to an architecture, and the figures illustrate the distribution functions computed on a pool of $30$ neural networks trained on the task.
• Figure 5: Neurons Strength and Activation, and scatter-plot of the correlation between Nodes Strength and Neurons Strength and Activation, for seven-layer depth FCs and AEs. The figures illustrate the distribution functions computed on a pool of $30$ neural networks trained on the task.