Understanding Visual Feature Reliance through the Lens of Complexity

TL;DR

This work introduces a new metric for quantifying feature complexity, based on $\mathscr{V}$-information and capturing whether a feature requires complex computational transformations to be extracted, and finds that complex features tend to be less important.

Abstract

Recent studies suggest that deep learning models inductive bias towards favoring simpler features may be one of the sources of shortcut learning. Yet, there has been limited focus on understanding the complexity of the myriad features that models learn. In this work, we introduce a new metric for quantifying feature complexity, based on $\mathscr{V}$-information and capturing whether a feature requires complex computational transformations to be extracted. Using this $\mathscr{V}$-information metric, we analyze the complexities of 10,000 features, represented as directions in the penultimate layer, that were extracted from a standard ImageNet-trained vision model. Our study addresses four key questions: First, we ask what features look like as a function of complexity and find a spectrum of simple to complex features present within the model. Second, we ask when features are learned during training. We find that simpler features dominate early in training, and more complex features emerge gradually. Third, we investigate where within the network simple and complex features flow, and find that simpler features tend to bypass the visual hierarchy via residual connections. Fourth, we explore the connection between features complexity and their importance in driving the networks decision. We find that complex features tend to be less important. Surprisingly, important features become accessible at earlier layers during training, like a sedimentation process, allowing the model to build upon these foundational elements.

Figures (18)

• Figure 1: A) Simple vs. Complex Features. Shown is an example of three features extracted using an overcomplete dictionary on the penultimate layer of a ResNet50 trained on ImageNet. Although all three features can be extracted from the final layer of a ResNet50, some features, such as $\textcolor{blue}{\mathstat{z}_1}$, seem to respond to color, which can be linearly extractable directly from the input. In contrast, $\textcolor{blue}{\mathstat{z}_2, \mathstat{z}_3}$ visualization appear more "Complex", responding to more diverse stimuli. In this work, we seek to study the complexity of features. We start by introducing a computationally inspired complexity metric. Using this metric, we inspect both simple and complex features of a ResNet50. B) Feature Evolution Across Layers. Each row illustrates how a feature from the penultimate layer ($\textcolor{blue}{\mathstat{z}_1, \mathstat{z}_2, \mathstat{z}_3}$) evolves as we decode it using linear probing at the outputs of blocks 1, 5, and 10 of the ResNet50. Simpler features, like color, are decodable throughout the network. The feature in the middle shows similar visualization at block 10 and the penultimate layer, whereas the most complex feature is only decodable at the end. Our complexity metric, based on $\mathcal{V}$-information xu2019theory, measures how easily a model extracts a feature across its layers.
• Figure 2: Qualitative Analysis of "Meta-feature" (cluster of features) Complexity.(Left) A 2D UMAP projection displaying the 10,000 extracted features. The features are organized into 150 clusters using K-means clustering applied to the feature dictionary $\mathstatbold{D}^\star$. 30 clusters were selected for analysis of features at different complexity levels. (Right) For each Meta-feature cluster, we compute the average complexity score. This allows us to classify the features based on their complexity according to the model. Notably, simple features are often akin to color detectors (e.g., grass, sky) and detectors for low-frequency patterns (e.g., bokeh detector) or lines. In contrast, complex features encompass parts or structured objects, as well as features resembling shapes (such as ears or curve detectors). Visualizations of individual Meta-features are presented in Appendix \ref{['ap:viz']}.
• Figure 3: Visualization of Meta-features, sorted by Complexity. We use Feature visualization olah2017featurefel2023unlocking to visualize the Meta-features found after concept extraction. The entire visualization for each Meta-feature can be found in Appendix \ref{['ap:viz']}.
• Figure 4: Simple Features Teleported by Residuals.(Left) CKA between residual branch activations $\bm{f}_\ell$ and final concept value $\mathstat{z}$. For simple concepts, beyond a certain layer (block 3), the residual already carries nearly all the information, effectively teleporting it to the last layer. (Right) Conversely, for complex features, both the main and residual branches gradually construct the features during the forward pass.
• Figure 5: A) Complex features emerge later in training. There is a strong correlation between the complexity of a feature and the requisite temporal span for its decoding. The temporal decoding score, $\Lambda$, is derived as the mean $\mathcal{V}$-information across epochs, with $\mathcal{V}$ representing the class encompassing linear models. A low score indicates a feature is accessible earlier during the training continuum, whereas a high score implies its tardy availability. The correlation between these scores suggests that complex features tend to emerge later in training. B) Important features are being compressed by the neural network: Levin Machine hypothesis. The average complexity of 10,000 features extracted independently at each epoch increases rapidly before stabilizing (the black curve shows the average). However, among the top-1% of features in terms of importance, complexity decreases over time, as if the model is self-compressing or simplifying, akin to a sedimentation process.