k* Distribution: Evaluating the Latent Space of Deep Neural Networks using Local Neighborhood Analysis

TL;DR

The paper tackles the problem that traditional latent-space visualizations based on dimensionality reduction distort local neighborhood structure, complicating class-wise interpretation. It introduces the k* distribution, where $k^*$ is the index of the nearest neighbor from a different class in the latent space, and summarizes per-class distributions with $\mu_{k^*}$, $\sigma_{k^*}$, and $\gamma_{k^*}$ to reveal three patterns: Pattern A (Fractured), Pattern B (Overlapped), and Pattern C (Clustered). The approach is model-agnostic and is validated through extensive experiments across architectures, network layers, training data distributions, adversarial robustness, and input transformations, with extensions to NLP and speech tasks. Key findings show that higher $\mu_{k^*}$ and lower $\gamma_{k^*}$ correlate with better accuracy, adversarial training tends to fracture latent space, and input perturbations consistently fragment local neighborhoods, illustrating the utility of k* distribution as a diagnostic tool for latent-space structure and robustness across domains.

Abstract

Most examinations of neural networks' learned latent spaces typically employ dimensionality reduction techniques such as t-SNE or UMAP. These methods distort the local neighborhood in the visualization, making it hard to distinguish the structure of a subset of samples in the latent space. In response to this challenge, we introduce the {k*~distribution} and its corresponding visualization technique This method uses local neighborhood analysis to guarantee the preservation of the structure of sample distributions for individual classes within the subset of the learned latent space. This facilitates easy comparison of different k*~distributions, enabling analysis of how various classes are processed by the same neural network. Our study reveals three distinct distributions of samples within the learned latent space subset: a) Fractured, b) Overlapped, and c) Clustered, providing a more profound understanding of existing contemporary visualizations. Experiments show that the distribution of samples within the network's learned latent space significantly varies depending on the class. Furthermore, we illustrate that our analysis can be applied to explore the latent space of diverse neural network architectures, various layers within neural networks, transformations applied to input samples, and the distribution of training and testing data for neural networks. Thus, the k* distribution should aid in visualizing the structure inside neural networks and further foster their understanding. Project Website is available online at https://shashankkotyan.github.io/k-Distribution/.

Figures (15)

• Figure 1: Overview of three distinct basic patterns of k* Distribution. Here, we define the k* value of a sample point as the kth-closest neighbor, which differs in class compared to the test point, i.e., the neighbor (sample) which breaks homogeneity in the local neighborhood of the test point. Pattern A ($\bigstar$) which has positively skewed k* distribution (skewed towards low k* value) representing an 'Fractured' distribution of samples in latent space; Pattern B ($\clubsuit$) which has almost uniform k* distribution representing a 'Overlapped' distribution of samples in latent space; Pattern C ($\spadesuit$) which has negatively skewed k* distribution (skewed towards high k* value) representing a 'Clustered' distribution of samples in latent space.
• Figure 2: Overview of the framework to create k* Distribution. We use the learned features of a neural network to compute k* values of individual evaluated samples and then compute the k* distribution for a particular class.
• Figure 3: Illustration of calculating k* value of a sample and correspondingly k* distribution of class. For all the samples, in the evaluating data, first find the index of the nearest neighbor that differs in class, i.e., where the local homogeneity of the neighbors breaks. We call this as k* value of the sample. Further, one can gather k* values for all the samples belonging to a single class and plot its distribution. In our example, note that the distribution of k* values for all the samples of yellow class is almost uniform, corresponding to Pattern B ($\clubsuit$)Overlapped distribution of samples in latent space. Similarly, the distribution of k* values for all the samples of green is positively skewed, which corresponds to Pattern A ($\bigstar$)Fractured distribution of samples in latent space, and the distribution of k* values for all the samples of pink is negatively skewed which corresponds to Pattern C ($\spadesuit$)Clustered distribution of samples in latent space.
• Figure 4: Visualization of the distribution of samples in latent space using, (Left) k* distribution, and (Right) Dimensionality Reduction techniques like t-SNE (Top Left), Isomap (Top Right), PCA (Bottom Left), and UMAP (Bottom Right) of all classes of 16-class-ImageNet for the Logit Layer of ResNet-50 Architecture. Note that the distribution of samples for a particular class is easier to compare using k* distribution than dimensionality reduction techniques.
• Figure 5: We visualize the neighbor distribution of all samples of a class for ResNet-50 architecture he2016deep (see Table \ref{['tab:architectures']}). The green color represents that the neighbor to the sample belongs to the same class as the testing sample, while the gray color represents that the neighbor belongs to a different class compared to the testing sample. A Fractured distribution of samples will have different class neighbors above the diagonal (black dashed line); An Overlapped distribution of samples will first different class neighbors around the diagonal, and; A Clustered distribution of samples will have different class neighbors below the diagonal.