Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity

TL;DR

This work investigates Transformer behavior through entropy and dynamic sparsity in a controlled setting with a predefined target distribution $p_{ m tgt}$. It reveals that Transformers favor learning low-entropy distributions $p_{ heta}$ (i.e., $H(p_{ heta})$ is often smaller than $H(p_{ m tgt})$) as model size grows, a bias largely driven by the FFN module and accompanied by parameter redundancy via dynamic sparsity. The study further shows that larger Transformers increasingly bypass attention computations using residual paths and exhibit stronger FFN-driven dynamic sparsity, with training loss spikes tightly linked to sudden drops in active neurons and influenced by second-order gradient information. These findings deepen our understanding of Transformer compression and efficiency, suggesting architectural and optimization dynamics that shape information content and computation at scale.

Abstract

Compression has been a critical lens to understand the success of Transformers. In the past, we have typically taken the target distribution as a criterion to evaluate a model's compression performance. Nevertheless,it often remains challenging to precisely assess how well the model achieves compression and to compare the information content of the learned distribution with that of the target distribution during compression,as the target distribution is typically unknown and entropy computation often incurs exponential cost. In this work, we explore these issues under a controlled experimental setup. We find that Transformers exhibit a unique inductive bias in data compression: beyond approaching the target distribution, they tend to favor learning lower-entropy distributions, with this tendency becoming more pronounced as the model size increases. This preference prevents Transformers from perfectly aligning with the target distribution, instead further compressing its information content. Furthermore, we show that the FFN module plays a critical role in driving this bias. In addition, while models remove informational redundancy from data during compression, they also exhibit redundancy within their parameters, which enables compression and can be characterized through dynamic sparsity. However, the dynamic sparsity patterns in Transformers, particularly in attention and FFN modules, demand further exploration. As for this, we show that larger Transformers show stronger preferences for bypassing attention computations via residual connections and have lower proportion of active neurons. Interestingly, we also find that training instability in larger models strongly correlates with sudden increases in dead neurons. Our work contributes to a deeper understanding of Transformers from the lens of entropy and dynamic sparsity.

Figures (21)

• Figure 1: Left: An example of generating a target distribution ${\bm{p}}_{{\bm{\theta}}}$ when $|{\mathcal{V}}| = 3$ and length $n = 3$. We choose the vocabulary $V = \{a, b, c\}$. We produce conditional probabilities to generate the final $p({\bm{s}})$. We introduce "#" as the the start symbol ${\bm{s}}_0$ and we define $p({\bm{s}}_0) = p$(#) = 1. First, ${\bm{p}}({\bm{s}}_1 |{\bm{s}}_0)$ is generated from a uniform distribution. Then, when generating the subsequent ${\bm{p}}({\bm{s}}_i | {\bm{s}}_{<i})$ for $i\geq 2$, we randomly select two characters and assign them transition probabilities of 0.8 and 0.2, while the remaining characters have a probability of 0. Right: The sequence ${\bm{s}}_{<i} = {\bm{s}}_0{\bm{s}}_1\dots{\bm{s}}_{i-1}$ is fed into the model to generate ${\bm{p}}(s_i | s_{<i})$ and $p_{{\bm{\theta}}}({\bm{s}})$ can be calculated as $p_{\bm{\theta}}({\bm{s}}) = \prod_{i=1}^{n} p_{{\bm{\theta}}}({\bm{s}}_i|{\bm{s}}_{<i})$.
• Figure 2: Left: Entropy and KL During Training for Transformers of Different Sizes. Center Left: Entropy and KL During Training for Transformer and LSTM when $d=8$ and $d=64$. Center Right: The change in entropy with model size for GRU, LSTM and Transformer, averaged over the last 15 epochs. Right: The relationship between entropy and KL for different model configurations (with each configuration averaged over the last 15 epochs).
• Figure 3: The change in entropy of the sparse and non-sparse parts with model size.
• Figure 4: Left: Entropy and KL during training for different Transformer variants. Center Left and Center Right: Relationship between KL/Loss and Entropy for different variants. FFN-main can achieve lower entropy compared to Attention-only and Attention-main. Right: Relationship between KL and Loss for different variants.
• Figure 5: Explanation of dynamic sparsity in the attention and FFN modules of Transformers. As for the Attention modules, each attention head or residual connection can be viewed as a forward path and larger Transformers prefer residual connections to bypass attention head computations. As for the FFN modules, the parameters of the first and second layer can be viewed as key and value vectors respectively and each neuron is a key-value pair. Larger Transformers will have a lower proportion of active neurons (i.e., key-value pairs).