SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground?

TL;DR

This work tackles the lack of explicit foreground–background separation in Vision Transformers by introducing SVD-ViT, which uses singular value decomposition to emphasize foreground information through a dedicated SPC token. The SPC module projects intermediate features onto a leading subspace to create foreground-focused aggregation tokens, and two plug-ins SSVA and ID-RSVD further refine this process by adaptively selecting and guiding the subspace. Empirical results on five image classification benchmarks show consistent improvements over ViT baselines, with notable gains when deeper SPC insertions are used and when combining SPC with SSVA and ID-RSVD. The findings suggest that spectral analysis of feature maps can yield robust foreground representations and open new avenues for incorporating SVD into deep feature extraction.

Abstract

Vision Transformers (ViT) have been established as large-scale foundation models. However, because self-attention operates globally, they lack an explicit mechanism to distinguish foreground from background. As a result, ViT may learn unnecessary background features and artifacts, leading to degraded classification performance. To address this issue, we propose SVD-ViT, which leverages singular value decomposition (SVD) to prioritize the learning of foreground features. SVD-ViT consists of three components-\textbf{SPC module}, \textbf{SSVA}, and \textbf{ID-RSVD}-and suppresses task-irrelevant factors such as background noise and artifacts by extracting and aggregating singular vectors that capture object foreground information. Experimental results demonstrate that our method improves classification accuracy and effectively learns informative foreground representations while reducing the impact of background noise.

Figures (6)

• Figure 1: Visualization of the leading left singular vectors obtained by applying SVD to the patch feature matrix (number of patches $\times$ embedding dimension) at each ViT layer. Each left singular vector is reshaped to the patch grid and rendered as a heatmap. From left to right, we show the input image, Layer 1, and Layer 10; for Layer 1 and Layer 10, the first to third singular vectors (e.g., #1--#3) are displayed. In shallow layers, components corresponding to the foreground emerge, whereas in deeper layers, spike-like high-norm components are observed.
• Figure 2: Overview of RSVD. A low-rank approximation matrix is constructed via randomized sketching and iterative orthogonalization, and applying SVD to the resulting matrix enables extracting only the leading singular vectors.
• Figure 3: Overview of SVD-ViT.
• Figure 4: Overview of SSVA.
• Figure 5: Overview of ID-RSVD.