Barcodes as Summary of Loss Function Topology

TL;DR

This work introduces a topological data analysis framework for neural network loss landscapes by using persistence barcodes of Morse complexes to summarize gradient-flow topology. It defines minima–1-saddle pairings and their associated birth–death segments, and provides a graph-based algorithm with $O(N\log N)$ complexity to compute the minima barcodes from point clouds. The authors demonstrate convergence on a suite of benchmark functions and apply the method to small neural networks, revealing that minima barcodes lie in a small lower region of the loss range and move lower as network depth and width grow, with implications for learning dynamics and generalization. The approach offers a scalable, geometry-driven perspective on loss surfaces and suggests pathways to extend to larger architectures and to connect topology with optimization behavior and generalization performance.

Abstract

We propose to study neural networks' loss surfaces by methods of topological data analysis. We suggest to apply barcodes of Morse complexes to explore topology of loss surfaces. An algorithm for calculations of the loss function's barcodes of local minima is described. We have conducted experiments for calculating barcodes of local minima for benchmark functions and for loss surfaces of small neural networks. Our experiments confirm our two principal observations for neural networks' loss surfaces. First, the barcodes of local minima are located in a small lower part of the range of values of neural networks' loss function. Secondly, increase of the neural network's depth and width lowers the barcodes of local minima. This has some natural implications for the neural network's learning and for its generalization properties.

Figures (6)

• Figure 1: Merging of connected components of sublevel sets at 1-saddles. (a)"Death" of the connected component $S_3$, the connected component $S_3$ of sublevel set merges with connected component $S_2$ at 1-saddle $q_3$, 1-saddle $q_3$ is associated with the minimum $p_3$. (b)"Death" of the connected component $S_4$, the connected component $S_4$ of sublevel set merges with connected component $S_1$ at 1-saddle $q_4$, 1-saddle $q_4$ is associated with the minimum $p_4$. (c)"Death" of the connected component $S_2$, the connected component $S_2$ of sublevel set merges with connected component $S_1$ at 1-saddle $q_2$, 1-saddle $q_2$ is associated with the minimum $p_2$. Note that the 1-saddle $q_2$ is associated with the minimum $p_2$ which is separated by another minimum from the green saddle.
• Figure 2: HumpCamel6 function, its minima-saddle correspondence and barcode computed by Algorithm \ref{['algorithm-graph']}.
• Figure 3: Langermann function, its minima-saddle correspondence and barcodes computed by Algorithm \ref{['algorithm-graph']}.
• Figure 6: Decrease of logarithm of Bottleneck distance between pairs of persistent diagrams on samples of size $N$ for Alpine01, Schwefel26, XinSheYang04 benchmark functions in dimensions $D\in \{3,4,5,6\}$.
• Figure 7: Barcodes quantify topological obstacles posed by local minima for gradient-based optimization. Here the two local minima locally look the same but clearly pose different obstacles for gradient-based learning. These obstacles are quantified by the lengths of green segments, which are associated with the local minima in the barcode.

Theorems & Definitions (14)

• Definition 1
• Proposition 1
• proof
• Proposition 2
• proof
• Theorem 3
• Definition 2
• Proposition 4
• proof
• Remark 1