Topological Signatures of ReLU Neural Network Activation Patterns

TL;DR

This work investigates topological signatures of ReLU-based feedforward networks by analyzing the polyhedral input-space decomposition and its dual graph. It demonstrates that a vertex-weighted Fiedler partition of the dual graph aligns with binary decision boundaries during grokking, and that Betti-number dynamics from a random filtration of the activation-cell complex correlate with training loss in a regression setting using a Duffing PINN. Through two 2D classification experiments (Two Circles and Two Moons), the authors show that weighting by training-point counts yields near-perfect alignment with labels, while Betti-curve and heat-map analyses reveal how the network’s geometric decomposition reorganizes during learning. The study also discusses scalability limitations and outlines future directions, including boundary-aware filtrations and higher-dimensional topology, to provide topology-based diagnostics for learning dynamics and generalization.

Abstract

This paper explores the topological signatures of ReLU neural network activation patterns. We consider feedforward neural networks with ReLU activation functions and analyze the polytope decomposition of the feature space induced by the network. Mainly, we investigate how the Fiedler partition of the dual graph and show that it appears to correlate with the decision boundary -- in the case of binary classification. Additionally, we compute the homology of the cellular decomposition -- in a regression task -- to draw similar patterns in behavior between the training loss and polyhedral cell-count, as the model is trained.

Figures (10)

• Figure 1: Example of polyhedral decomposition of the domain of a FFRNN. Grayed out activation function outputs correspond to ReLU value of zero at that neuron, indicating a 0 entry in the binary state vector.
• Figure 2: The two circles and two moons datasets (left column), their respective polytope decompositions (middle left), and the partitions of the dual graph by the unweighted (middle right) and weighted Fiedler vectors (right), indicated by the blue and purple points. Minimal loss and concentration along decision boundary demonstrates grokking behavior.
• Figure 3: (Left) Prediction of a FFRNN trained to learn a paraboloid surface. (Middle) Snapshots of a random filtration on the primal complex of the polyhedral decomposition of such a network with their corresponding Betti numbers. First, randomly add all the vertices of the decomposition (blue), then add the edges (red, cycles are born), and finally add the faces (green, cycles are closed). (Right) Dual graph of the polyhedral decomposition: each polytope is now a vertex, and join two vertices if their corresponding polytopes share a face.
• Figure 4: (Left) Snapshots (zoomed in to the square $[0.3,0.3]^2$) of the polyhedral decomposition of a PINN at the beginning (epoch 0) and end of training (epoch 10000). In red are the vertices (0-cells), in solid blue the edges ($1$-cells), and in a lighter blue the faces ($2$-cells). More cells are observed at the beginning of training. This number seems to decrease as we train, with the emergence of some structure. (Right) Corresponding average Betti curves, $\beta_0$ and $\beta_1$, of each stage of training for a random filtration of the resulting complex.
• Figure 5: Heat maps of the Betti curves corresponding to a trained PINN, with the training loss overlaid. (Left) For $\beta_0$ we observe smoother changes. (Right) For $\beta_1$, the changes are sharper and correlate with substantial changes in the training loss function.