Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multistability of Self-Attention Dynamics in Transformers

Claudio Altafini

TL;DR

The paper analyzes a continuous-time self-attention model for transformers, reframing it as a multiagent dynamical system on the unit sphere and linking it to a multiagent Oja flow that targets the principal eigenvector of the value matrix $V$. It provides a rigorous classification of equilibria—consensus, bipartite consensus, clustering, and polygonal—and derives stability conditions, showing that multistability is common and that many stable states align with eigenvectors of $V$, often the principal one. Through theoretical results and numerical experiments, it demonstrates that self-attention dynamics can converge to low-rank attractors and that attention weighting introduces rich bifurcation behavior beyond the classic Oja flow. The findings offer a nonlinear Perron–Frobenius perspective on transformer layers, suggesting that successive layers may tilt token representations toward eigenvectors of $V$ and inviting experimental validation on pretrained models.

Abstract

In machine learning, a self-attention dynamics is a continuous-time multiagent-like model of the attention mechanisms of transformers. In this paper we show that such dynamics is related to a multiagent version of the Oja flow, a dynamical system that computes the principal eigenvector of a matrix corresponding for transformers to the value matrix. We classify the equilibria of the ``single-head'' self-attention system into four classes: consensus, bipartite consensus, clustering and polygonal equilibria. Multiple asymptotically stable equilibria from the first three classes often coexist in the self-attention dynamics. Interestingly, equilibria from the first two classes are always aligned with the eigenvectors of the value matrix, often but not exclusively with the principal eigenvector.

Multistability of Self-Attention Dynamics in Transformers

TL;DR

The paper analyzes a continuous-time self-attention model for transformers, reframing it as a multiagent dynamical system on the unit sphere and linking it to a multiagent Oja flow that targets the principal eigenvector of the value matrix . It provides a rigorous classification of equilibria—consensus, bipartite consensus, clustering, and polygonal—and derives stability conditions, showing that multistability is common and that many stable states align with eigenvectors of , often the principal one. Through theoretical results and numerical experiments, it demonstrates that self-attention dynamics can converge to low-rank attractors and that attention weighting introduces rich bifurcation behavior beyond the classic Oja flow. The findings offer a nonlinear Perron–Frobenius perspective on transformer layers, suggesting that successive layers may tilt token representations toward eigenvectors of and inviting experimental validation on pretrained models.

Abstract

In machine learning, a self-attention dynamics is a continuous-time multiagent-like model of the attention mechanisms of transformers. In this paper we show that such dynamics is related to a multiagent version of the Oja flow, a dynamical system that computes the principal eigenvector of a matrix corresponding for transformers to the value matrix. We classify the equilibria of the ``single-head'' self-attention system into four classes: consensus, bipartite consensus, clustering and polygonal equilibria. Multiple asymptotically stable equilibria from the first three classes often coexist in the self-attention dynamics. Interestingly, equilibria from the first two classes are always aligned with the eigenvectors of the value matrix, often but not exclusively with the principal eigenvector.

Paper Structure

This paper contains 13 sections, 14 theorems, 32 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Model formulation
  3. Oja flow
  4. Multiagent Oja flow
  5. Equilibria
  6. Stability analysis
  7. Self-Attention dynamics
  8. Equilibria
  9. Stability analysis
  10. Numerical examples
  11. Extensions of the model
  12. Conclusion
  13. Acknowledgments

Key Result

Lemma 1

All eigenvectors $\bm{v}_k$ of $V$ (more precisely, the values $\pm \bm{v}_k$, $\| \bm{v}_k\|=1$) are equilibria of eq:oja1. The function $W(\bm{x}) = \frac{1}{2} (\lambda_1- R(\bm{x}))$ is a Lyapunov function for eq:oja1 and guarantees that eq:oja1 converges to the principal eigenvector $\pm \bm{v}

Figures (3)

  • Figure 1: Example \ref{['ex:3dgeneral-1']}, with $d=3$ and $n=10$. The solid dot is the endpoint of a trajectory.
  • Figure 2: Example \ref{['ex:equilibria-1']}, numerical classification of stable equilibria.
  • Figure 3: Example \ref{['ex:equilibria-2']}. (a): exhaustive count of all stable consensus + bipartite consensus equilibria; (b): random sampling stable bipartite consensus equilibria.

Theorems & Definitions (23)

  • Lemma 1
  • Remark 1
  • Lemma 2
  • Theorem 1
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Remark 2
  • Lemma 6
  • Remark 3
  • ...and 13 more