Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes

Yingyi Chen, Qinghua Tao, Francesco Tonin, Johan A. K. Suykens

TL;DR

The paper tackles the challenge of uncertainty estimation in Transformer-based self-attention by treating the attention kernel as inherently asymmetric. It introduces Kernel-Eigen Pair Sparse Variational Gaussian Processes (KEP-SVGP), which uses Kernel SVD to build a paired SVGP model based on adjoint eigenfunctions, enabling efficient posterior inference via a diagonal singular-value matrix. The approach yields improved uncertainty calibration, robustness to distribution shifts, and competitive OOD detection while maintaining scalable complexity. Empirical results across vision and language benchmarks demonstrate practical impact for uncertainty-aware Transformers with reduced computational cost.

Abstract

While the great capability of Transformers significantly boosts prediction accuracy, it could also yield overconfident predictions and require calibrated uncertainty estimation, which can be commonly tackled by Gaussian processes (GPs). Existing works apply GPs with symmetric kernels under variational inference to the attention kernel; however, omitting the fact that attention kernels are in essence asymmetric. Moreover, the complexity of deriving the GP posteriors remains high for large-scale data. In this work, we propose Kernel-Eigen Pair Sparse Variational Gaussian Processes (KEP-SVGP) for building uncertainty-aware self-attention where the asymmetry of attention kernels is tackled by Kernel SVD (KSVD) and a reduced complexity is acquired. Through KEP-SVGP, i) the SVGP pair induced by the two sets of singular vectors from KSVD w.r.t. the attention kernel fully characterizes the asymmetry; ii) using only a small set of adjoint eigenfunctions from KSVD, the derivation of SVGP posteriors can be based on the inversion of a diagonal matrix containing singular values, contributing to a reduction in time complexity; iii) an evidence lower bound is derived so that variational parameters and network weights can be optimized with it. Experiments verify our excellent performances and efficiency on in-distribution, distribution-shift and out-of-distribution benchmarks.

Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes

TL;DR

The paper tackles the challenge of uncertainty estimation in Transformer-based self-attention by treating the attention kernel as inherently asymmetric. It introduces Kernel-Eigen Pair Sparse Variational Gaussian Processes (KEP-SVGP), which uses Kernel SVD to build a paired SVGP model based on adjoint eigenfunctions, enabling efficient posterior inference via a diagonal singular-value matrix. The approach yields improved uncertainty calibration, robustness to distribution shifts, and competitive OOD detection while maintaining scalable complexity. Empirical results across vision and language benchmarks demonstrate practical impact for uncertainty-aware Transformers with reduced computational cost.

Abstract

While the great capability of Transformers significantly boosts prediction accuracy, it could also yield overconfident predictions and require calibrated uncertainty estimation, which can be commonly tackled by Gaussian processes (GPs). Existing works apply GPs with symmetric kernels under variational inference to the attention kernel; however, omitting the fact that attention kernels are in essence asymmetric. Moreover, the complexity of deriving the GP posteriors remains high for large-scale data. In this work, we propose Kernel-Eigen Pair Sparse Variational Gaussian Processes (KEP-SVGP) for building uncertainty-aware self-attention where the asymmetry of attention kernels is tackled by Kernel SVD (KSVD) and a reduced complexity is acquired. Through KEP-SVGP, i) the SVGP pair induced by the two sets of singular vectors from KSVD w.r.t. the attention kernel fully characterizes the asymmetry; ii) using only a small set of adjoint eigenfunctions from KSVD, the derivation of SVGP posteriors can be based on the inversion of a diagonal matrix containing singular values, contributing to a reduction in time complexity; iii) an evidence lower bound is derived so that variational parameters and network weights can be optimized with it. Experiments verify our excellent performances and efficiency on in-distribution, distribution-shift and out-of-distribution benchmarks.
Paper Structure (52 sections, 1 theorem, 52 equations, 5 figures, 9 tables)

This paper contains 52 sections, 1 theorem, 52 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Uncertainty Estimation
  4. Deep Gaussian Processes
  5. Bayesian Transformers
  6. Background
  7. Sparse Variational Gaussian Processes
  8. Gaussian Processes
  9. Sparse Variational Gaussian Processes
  10. Kernel-Eigen Features for SVGPs
  11. Self-Attention as Asymmetric Kernel Machine
  12. Self-Attention corresponds to Asymmetric Kernel
  13. Self-Attention with Kernel SVD
  14. KEP-SVGP for Self-Attention
  15. Kernel-Eigen Pair SVGP
  16. ...and 37 more sections

Key Result

Proposition 4.1

The Kullback–Leibler divergence in the ELBO objective eq::elbo_spg is equal to where $\Lambda\in\mathbb{R}^{s\times s}$ is diagonal whose inversion is of $\mathcal{O}(s)$.

Figures (5)

  • Figure 1: Illustration of canonical self-attention and our KEP-SVGP in one layer. (a) The attention kernel $K_{\rm att}$ in canonical self-attention is induced by two different feature maps $\phi_q,\phi_k$ related to queries and keys; hence $K_{\rm att}$ is in essence asymmetric. (b) KEP-SVGP consists of one SVGP pair induced by the two sets of projection outputs based on $\phi_q,\phi_k$ from KSVD w.r.t. $K_{\rm att}$, which fully characterizes the asymmetry of self-attention in the posterior. The posteriors are now approximated based on the inversion of a diagonal matrix $\Lambda$ containing top $s$ singular values, thereby of time complexity $\mathcal{O}(s)$.
  • Figure 2: Comparisons of our KEP-SVGP with MSP under distribution shift. Performance on 15 types of corruption under the severity level of 5 is reported, where models are trained on CIFAR-10/ViT and tested on CIFAR-10-C.
  • Figure 3: KEP-SVGP leads to better confidence separation between ID correct and OOD samples.
  • Figure 4: Spectrum analysis of the self-attention kernel matrix on CoLA. Specifically, we consider the normalized cumulative singular values of the attention matrix of the two-layer Transformer models. (a) plots the spectrum results of the first-layer kernel matrix; (b) plots the spectrum results of the second-layer kernel matrix; (c) plots the normalized cumulative singular values w.r.t. singular value index of each layer, showing the low-rank property of the attention matrix of each model.
  • Figure 5: Comparisons of our KEP-SVGP with baselines under distribution shift. Mean AURC results of all 5 severity levels under 15 types of corruption are reported, where models are trained on CIFAR-10/ViT and tested on CIFAR-10-C.

Theorems & Definitions (2)

  • Remark 3.1
  • Proposition 4.1