Decoupling Semantic Similarity from Spatial Alignment for Neural Networks

TL;DR

This paper measures semantic similarity between input responses by formulating it as a set-matching problem and quantifies the superiority of semantic RSMs over spatio-semantic RSMs through image retrieval and by comparing the similarity between representations to the similarity between predicted class probabilities.

Abstract

What representation do deep neural networks learn? How similar are images to each other for neural networks? Despite the overwhelming success of deep learning methods key questions about their internal workings still remain largely unanswered, due to their internal high dimensionality and complexity. To address this, one approach is to measure the similarity of activation responses to various inputs. Representational Similarity Matrices (RSMs) distill this similarity into scalar values for each input pair. These matrices encapsulate the entire similarity structure of a system, indicating which input leads to similar responses. While the similarity between images is ambiguous, we argue that the spatial location of semantic objects does neither influence human perception nor deep learning classifiers. Thus this should be reflected in the definition of similarity between image responses for computer vision systems. Revisiting the established similarity calculations for RSMs we expose their sensitivity to spatial alignment. In this paper, we propose to solve this through semantic RSMs, which are invariant to spatial permutation. We measure semantic similarity between input responses by formulating it as a set-matching problem. Further, we quantify the superiority of semantic RSMs over spatio-semantic RSMs through image retrieval and by comparing the similarity between representations to the similarity between predicted class probabilities.

Figures (17)

• Figure 1: Current spatio-semantic RSMs couple semantic similarity with spatial alignment. Our proposal focuses solely on measuring semantic similarity. We achieve this by determining the optimal permutation between two representations and introducing sample-wise permutation invariance.
• Figure 2: Semantic RSMs capture similarity independent of spatial localization, in contrast to current spatio-semantic RSMs. We utilize Tiny-ImageNet to generate partially overlapping crops of the same sample (left) and calculate RSMs for a trained ResNet18 model. The plot displays the original spatio-semantic RSMs (middle top) and our proposed semantic RSMs (middle bottom) across various layers for a single batch. Additionally, the distribution of similarity values over multiple batches is shown (right). The results indicate that spatio-semantic RSMs struggle to detect largely identical but translated images, while semantic RSMs exhibit an enhanced off-diagonal in the RSMs and a significant gap between distributions. This demonstrates the capability of our method to detect the same semantics even when translated.
• Figure 3: Relaxing the constraint of spatial alignment leads to better retrieval. We leverage general feature extractors to embed images of the EgoObjects dataset. We then compare these embeddings either with or without permutation invariance. PI: Permutation Invariant
• Figure 4: Retrieving by permutation invariant similarity returns similar scenes of different spatial geometry. We visualize the top 3 most similar images according to two exemplary query images for SAM ViT/B32.
• Figure 5: Approximative algorithms yield comparable matching quality to optimal algorithms. The ratio of similarity from various approximations relative to maximal semantic similarity is visualized across multiple layers of a ResNet18.