Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Characterization of topological structures in different neural network architectures

Paweł Świder

TL;DR

This work addresses understanding neural network internals by applying Topological Data Analysis (TDA) to layer activations, using persistent homology ($H_k$) and Betti numbers ($\beta_k$) on Vietoris–Rips complexes to quantify topology across ResNet, VGG19, and ViT. The authors propose a practical workflow, investigate the effects of sample size and outliers on persistence diagrams (noting a threshold around a few hundred points and generally modest LOF impact), and compare topological features across architectures and layers, including finetuning effects. Key findings include architecture-dependent topology with deeper layers often undergoing stronger topological transformations, shared topological tendencies among similarly structured models, and ViT displaying distinctive, outlier-rich, and mid-to-late-layer divergence between pre-trained and finetuned representations. By demonstrating consistent train/test topology and outlining guidelines for fair diagram comparisons, the study validates TDA as a powerful tool for interpreting neural representations and guiding future topology-based analyses across diverse architectures.

Abstract

One of the most crucial tasks in the future will be to understand what is going on in neural networks, as they will become even more powerful and widely deployed. This work aims to use TDA methods to analyze neural representations. We develop methods for analyzing representations from different architectures and check how one should use them to obtain valid results. Our findings indicate that removing outliers does not have much impact on the results and that we should compare representations with the same number of elements. We applied these methods for ResNet, VGG19, and ViT architectures and found substantial differences along with some similarities. Additionally, we determined that models with similar architecture tend to have a similar topology of representations and models with a larger number of layers change their topology more smoothly. Furthermore, we found that the topology of pre-trained and finetuned models starts to differ in the middle and final layers while remaining quite similar in the initial layers. These findings demonstrate the efficacy of TDA in the analysis of neural network behavior.

Characterization of topological structures in different neural network architectures

TL;DR

This work addresses understanding neural network internals by applying Topological Data Analysis (TDA) to layer activations, using persistent homology () and Betti numbers () on Vietoris–Rips complexes to quantify topology across ResNet, VGG19, and ViT. The authors propose a practical workflow, investigate the effects of sample size and outliers on persistence diagrams (noting a threshold around a few hundred points and generally modest LOF impact), and compare topological features across architectures and layers, including finetuning effects. Key findings include architecture-dependent topology with deeper layers often undergoing stronger topological transformations, shared topological tendencies among similarly structured models, and ViT displaying distinctive, outlier-rich, and mid-to-late-layer divergence between pre-trained and finetuned representations. By demonstrating consistent train/test topology and outlining guidelines for fair diagram comparisons, the study validates TDA as a powerful tool for interpreting neural representations and guiding future topology-based analyses across diverse architectures.

Abstract

One of the most crucial tasks in the future will be to understand what is going on in neural networks, as they will become even more powerful and widely deployed. This work aims to use TDA methods to analyze neural representations. We develop methods for analyzing representations from different architectures and check how one should use them to obtain valid results. Our findings indicate that removing outliers does not have much impact on the results and that we should compare representations with the same number of elements. We applied these methods for ResNet, VGG19, and ViT architectures and found substantial differences along with some similarities. Additionally, we determined that models with similar architecture tend to have a similar topology of representations and models with a larger number of layers change their topology more smoothly. Furthermore, we found that the topology of pre-trained and finetuned models starts to differ in the middle and final layers while remaining quite similar in the initial layers. These findings demonstrate the efficacy of TDA in the analysis of neural network behavior.
Paper Structure (23 sections, 7 equations, 26 figures, 2 tables)

This paper contains 23 sections, 7 equations, 26 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Proposed experiments
  4. How we can use TDA for neural reprensetations
  5. How does the number of points impact persistent homology
  6. Impact of outliers on persistence homology
  7. Experiments on topology of neural representations
  8. Topological characterization of selected architectures
  9. Effects of finetuning on topology of representations
  10. Where networks change homology most rapidly
  11. Results of experiments
  12. How does the number of points impacts persistent homology
  13. Impact of outliers on persistence homology
  14. Topological characterization of selected architectures
  15. Detailed analysis of plain convolutional networks
  16. ...and 8 more sections

Figures (26)

  • Figure 1: Vietoris-Rips complexes on toy data, with the different $\varepsilon$ value.
  • Figure 2: Persistence diagram and persistence barcodes for data from \ref{['fig:methods-complex']}.
  • Figure 3: Two datasets with clean and noisy data along with persistence diagrams for $H_1$ homological features with bottleneck distance matching between them.
  • Figure 4: Violin plots of the homological features along with bottleneck distances calculated between diagrams obtained from a subset of points with all available for VGG19.
  • Figure 5: Violin plots of the homological features along with bottleneck distances calculated between diagrams obtained from a subset of points with all available for ResNet18.
  • ...and 21 more figures