Why Is Attention Sparse In Particle Transformer?

TL;DR

This work investigates why ParT's attention is sparsely distributed by comparing multiple jet-tagging datasets and disentangling the contributions of the attention mechanism versus the physics-inspired interaction matrix. Through pre-softmax analyses and ablations, the authors show that the sparse, nearly binary attention primarily arises from the attention term itself, with the interaction matrix providing secondary biases essential for full performance. They demonstrate that ParT can identify key jet substructure elements, including leptons in semileptonic top decays, even without explicit PID inputs, indicating substructure is encoded in early attention maps. The findings enhance interpretability of transformer-based jet taggers and suggest practical efficiency gains, such as adopting top-$k$ attention, while outlining limitations and directions for future work on physics-informed training and broader benchmarking.

Abstract

Transformer-based models have achieved state-of-the-art performance in jet tagging at the CERN Large Hadron Collider (LHC), with the Particle Transformer (ParT) representing a leading example of such models. A striking feature of ParT is its sparse, nearly binary, attention structure, raising questions about the origin of this behavior and whether it encodes physically meaningful correlations. In this work, we investigate the source of ParT's sparse attention by comparing models trained on multiple benchmark datasets and examine the relative contributions of the attention term and the physics-inspired interaction matrix before softmax. We find that binary sparsity arises primarily from the attention mechanism itself, with the interaction matrix playing a secondary role. Moreove, we show that ParT is able to identify key jet substructure elements, such as leptons in semileptonic top decays, even without explicit particle identification inputs. These results provide new insight into the interpretability of transformer-based jet taggers and clarify the conditions under which sparse attention patterns emerge in ParT.

Figures (2)

• Figure 1: Comparison of ParT attention distributions across datasets (a--d), $\eta$--$\phi$ particle attention maps (e--h), and the distribution of attention scores to the lepton in $t\to b \ell \nu$ with ParT trained on JetClass Kinematic (i). $\eta$--$\phi$ maps use pre-softmax attention values without the interaction matrix included. Symbols: (✖: muon, ▲: charged hadron, ▼: neutral hadron, ●: photon, ✚: electron). Transparency scales with particle $p_{\mathrm{T}}\xspace$, while line intensity reflects attention scores.
• Figure 2: Distribution of the magnitude of pre-softmax attention value divided by pre-softmax interaction matrix value. For both JetClass models, the ratio is almost always greater than one, and often $10^4$--$10^5$ times bigger than the interaction matrix, meaning it completely dominates.