Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Primal-Dual Framework for Transformers and Neural Networks

Tan M. Nguyen, Tam Nguyen, Nhat Ho, Andrea L. Bertozzi, Richard G. Baraniuk, Stanley J. Osher

TL;DR

This work demonstrates the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model's accuracy, and improving the model's efficiency in a variety of practical applications including image and time-series classification.

Abstract

Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision. Like neural network layers, these attention mechanisms are often developed by heuristics and experience. To provide a principled framework for constructing attention layers in transformers, we show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem, whose primal formulation has the form of a neural network layer. Using our framework, we derive popular attention layers used in practice and propose two new attentions: 1) the Batch Normalized Attention (Attention-BN) derived from the batch normalization layer and 2) the Attention with Scaled Head (Attention-SH) derived from using less training data to fit the SVR model. We empirically demonstrate the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model's accuracy, and improving the model's efficiency in a variety of practical applications including image and time-series classification.

A Primal-Dual Framework for Transformers and Neural Networks

TL;DR

This work demonstrates the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model's accuracy, and improving the model's efficiency in a variety of practical applications including image and time-series classification.

Abstract

Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision. Like neural network layers, these attention mechanisms are often developed by heuristics and experience. To provide a principled framework for constructing attention layers in transformers, we show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem, whose primal formulation has the form of a neural network layer. Using our framework, we derive popular attention layers used in practice and propose two new attentions: 1) the Batch Normalized Attention (Attention-BN) derived from the batch normalization layer and 2) the Attention with Scaled Head (Attention-SH) derived from using less training data to fit the SVR model. We empirically demonstrate the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model's accuracy, and improving the model's efficiency in a variety of practical applications including image and time-series classification.
Paper Structure (37 sections, 1 theorem, 49 equations, 4 figures, 14 tables)

This paper contains 37 sections, 1 theorem, 49 equations, 4 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Background: Self-Attention
  3. Contribution
  4. Primal-Dual Interpretation of Self-Attention
  5. Attention as a Support Vector Regression Model
  6. Deriving Popular Attention Mechanisms as the Support Vector Expansion
  7. Linear Attention
  8. Sparse Attention
  9. Multi-head Attention (MHA)
  10. Deriving New Attention Mechanisms: Batch Normalized Attention and Multiresolution Head Attention
  11. Batch Normalized Attention
  12. Attention with Scaled Heads
  13. Experimental Results
  14. Empirical Analysis
  15. Related Work
  16. ...and 22 more sections

Key Result

Theorem 1

Given the function $f$ defined in Eqn. eqn:primal-model with $h({\bm x}) := \sum_j^N\Phi({\bm x})^T\Phi({\bm k}_j)$ and the support vector regression problem defined in Eqn. eqn:primal-opt, we set ${\bm b}=0$, choose $\Phi({\bm x})$ as in Eqn. eqn:phix, and relax the boundness constraint of the vari

Figures (4)

  • Figure 1: (Left) FLOPS ratios and (Right) memory usage ratios between the Attention-BN+SH and the softmax baseline trained on retrieval task for different model dimensions and sequence lengths. The reduction in computation and memory when using our models improves with sequence length. When scaling up the model, our methods remain significantly more beneficial than the baseline.
  • Figure 2: (Left) FLOPS ratios and (Right) memory usage ratios between the Attention-SH and the softmax attention baseline trained on the LRA retrieval task for different model dimensions and sequence lengths.
  • Figure 3: (Left) FLOPS ratios and (Right) memory usage ratios between the Sparse Attention-BN+SH and the Sparse Attention baseline trained on the LRA retrieval task for different model dimensions and sequence lengths. When using our models, the reduction in computation and memory improves with sequence length. When scaling up the model with greater model dimension, our methods remain significantly more efficient than the baseline.
  • Figure 4: (Left) FLOPS ratios and (Right) memory usage ratios between the Sparse Attention-SH and the Sparse Attention baseline trained on the LRA retrieval task for different model dimensions and sequence lengths. When using our models, the reduction in computation and memory improves with sequence length. When scaling up the model with greater model dimension, our methods remain significantly more efficient than the baseline.

Theorems & Definitions (13)

  • Remark 1
  • Theorem 1: Softmax Attention as a Support Vector Expansion
  • Remark 2
  • Remark 3
  • Remark 4
  • Definition 1: Batch Normalized Attention
  • Definition 2: Attention with Scaled Heads
  • Remark 5
  • Remark 6
  • Definition 3: 2D-Convolution Attention
  • ...and 3 more