Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Transformer Encoder-Decoder Representations through Bernoulli Dropout

Xuanzhou Chen

TL;DR

The paper tackles Transformer overparameterization by analyzing angular structure in high-dimensional encoder–decoder embeddings and testing sparsity through Bernoulli dropout. It introduces a BEC-based dropout and AWGN perturbations, providing a theoretical guarantee that Top-1 predictions are preserved when the effective sparsity is large and the pre-dropout margin is positive; this is validated empirically on English–French translation showing a sharp performance breakdown threshold. The work highlights that high-dimensional semantic spaces carry redundancy that can be exploited to reduce computation without large losses in accuracy or BLEU, while also offering insights into semantic preservation under aggressive feature erasure. Overall, the findings point to potential for more parameter-efficient Transformer architectures guided by information-theoretic and geometric considerations.

Abstract

We study Transformer overparameterization through the lens of angular similarity in high-dimensional encoder-decoder embeddings. We apply Bernoulli dropout between the encoder and the decoder, varying the keep probability $p$ to identify a sparsity-dependent threshold above which the Top-1 prediction is preserved. Theoretically, we prove that, if the effective sparsity embeddings is sufficiently large, and thus decoder performance, remain stable under moderate coordinate dropout. Empirically, we implement the Bernoulli dropout by constructing a new Transformer model augmented with Binary Erasure Channel (BEC) and test its performance on an English-French translation task. Experimental results visualize the trends for validation accuracies and BLEU scores, both decline sharply at some threshold.

Understanding Transformer Encoder-Decoder Representations through Bernoulli Dropout

TL;DR

The paper tackles Transformer overparameterization by analyzing angular structure in high-dimensional encoder–decoder embeddings and testing sparsity through Bernoulli dropout. It introduces a BEC-based dropout and AWGN perturbations, providing a theoretical guarantee that Top-1 predictions are preserved when the effective sparsity is large and the pre-dropout margin is positive; this is validated empirically on English–French translation showing a sharp performance breakdown threshold. The work highlights that high-dimensional semantic spaces carry redundancy that can be exploited to reduce computation without large losses in accuracy or BLEU, while also offering insights into semantic preservation under aggressive feature erasure. Overall, the findings point to potential for more parameter-efficient Transformer architectures guided by information-theoretic and geometric considerations.

Abstract

We study Transformer overparameterization through the lens of angular similarity in high-dimensional encoder-decoder embeddings. We apply Bernoulli dropout between the encoder and the decoder, varying the keep probability to identify a sparsity-dependent threshold above which the Top-1 prediction is preserved. Theoretically, we prove that, if the effective sparsity embeddings is sufficiently large, and thus decoder performance, remain stable under moderate coordinate dropout. Empirically, we implement the Bernoulli dropout by constructing a new Transformer model augmented with Binary Erasure Channel (BEC) and test its performance on an English-French translation task. Experimental results visualize the trends for validation accuracies and BLEU scores, both decline sharply at some threshold.
Paper Structure (18 sections, 1 theorem, 15 equations, 5 figures, 2 tables)

This paper contains 18 sections, 1 theorem, 15 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Overparameterization
  4. High-dimensional representation
  5. Dropout inspired from information theory and learning theory
  6. Methodology
  7. Add AWGN noise
  8. Theoretical results
  9. Experiments
  10. Data pre-processing
  11. Experiment setup
  12. Numerical Results
  13. Training time and convergence
  14. Test accuracy and BLEU evaluation
  15. Semantic Results
  16. ...and 3 more sections

Key Result

Theorem 1

Let $q\in\mathbb{R}^d$ be the decoder state just before the output projection and $\{v_j\}_{j=1}^M\subset\mathbb{R}^d$ be unit output embeddings. Assume $\|q\|_2=1$ and define $s_j=\langle q,v_j\rangle$, $j^\star=\arg\max_{j\in[M]} s_j$, and the margin $\gamma:= s_{j^\star}-\max_{j\neq j^\star} s_j> Consequently, if $\gamma>2C \sqrt{\frac{\log(M/\delta)}{ps_{\mathrm{eff}}(q)}},$ then the post-drop

Figures (5)

  • Figure 1: BEC-augmented Transformer architecture: The BEC between the encoder and decoder applies the Bernoulli mask $\mathbf{m}_i$. During training, Additive White Gaussian Noise (AWGN) is applied to perturb the input embeddings.
  • Figure 2: Validation accuracy and Average BLEU (per sentence) across different Bernoulli dropout probabilities ($p$) under both noise-free and AWGN training settings.
  • Figure 3: Scheduled learning rate in Adam for noise-free and AWGN training.
  • Figure 4: (a) Training and testing accuracies for the BEC-augmented Transformer with dropout probability $0.2$ in the noise-free experiment. (b) Training and testing losses for the BEC-augmented Transformer with dropout probability $0.2$ in the noise-free experiment. (c) Training and testing accuracies for the BEC-augmented Transformer with dropout probability $0.5$ in the noise-free experiment. (d)Training and testing losses for the BEC-augmented Transformer with dropout probability $0.5$ in the noise-free experiment.
  • Figure 5: Attention visualization heatmap for BEC-augmented Transformer with dropout probability at $0.5$.

Theorems & Definitions (1)

  • Theorem 1: Top-1 prediction preserved under coordinate dropout, proved in Appendix \ref{['app:pf_thm']}