Interpretable Vision Transformers in Monocular Depth Estimation via SVDA

TL;DR

Monocular depth estimation with Vision Transformers suffers from opaque self-attention. The paper proposes SVDA, a spectral attention mechanism that uses row-wise $L_2$-normalized queries and keys and a diagonal spectral modulation $\Sigma$, resulting in $A = \mathrm{softmax}\left(\frac{Q \Sigma K^\top}{\sqrt{d_k}}\right)$. Integrated into the Dense Prediction Transformer (DPT), SVDA preserves accuracy on KITTI and NYU-v2 while adding a small parameter increase and modest runtime overhead. More importantly, SVDA provides six spectral indicators—spectral entropy, effective rank, angular alignment, selectivity index, spectral sparsity, and perturbation robustness—that quantify how attention organizes during training and across depths, enabling intrinsic interpretability. This diagnostic framework supports cross-dataset comparisons and moves attention from a black-box mechanism to a principled descriptor for transparent dense prediction.

Abstract

Monocular depth estimation is a central problem in computer vision with applications in robotics, AR, and autonomous driving, yet the self-attention mechanisms that drive modern Transformer architectures remain opaque. We introduce SVD-Inspired Attention (SVDA) into the Dense Prediction Transformer (DPT), providing the first spectrally structured formulation of attention for dense prediction tasks. SVDA decouples directional alignment from spectral modulation by embedding a learnable diagonal matrix into normalized query-key interactions, enabling attention maps that are intrinsically interpretable rather than post-hoc approximations. Experiments on KITTI and NYU-v2 show that SVDA preserves or slightly improves predictive accuracy while adding only minor computational overhead. More importantly, SVDA unlocks six spectral indicators that quantify entropy, rank, sparsity, alignment, selectivity, and robustness. These reveal consistent cross-dataset and depth-wise patterns in how attention organizes during training, insights that remain inaccessible in standard Transformers. By shifting the role of attention from opaque mechanism to quantifiable descriptor, SVDA redefines interpretability in monocular depth estimation and opens a principled avenue toward transparent dense prediction models.

Figures (2)

• Figure 1: Validation loss over epochs for baseline and SVDA models across datasets. The models show nearly identical learning trajectories.
• Figure 2: SVDA interpretability diagnostics across datasets. Each row corresponds to one indicator: (a) Spectral Entropy, (b) Effective Rank, (c) Angular Alignment, (d) Selectivity Index, (e) Spectral Sparsity, (f) Perturbation Robustness. For each dataset block (KITTI on the left, NYU-v2 on the right), the first column shows evolution across training epochs and the second column shows the corresponding per-layer distribution (box-plots).