Bumblebee: Foundation Model for Particle Physics Discovery

TL;DR

The paper addresses permutation-invariant particle-physics event data by removing positional encodings and embedding 4-vectors, enabling joint modeling of generator and reconstruction information. It introduces Bumblebee, a BERT-inspired transformer with a Cloze-like pre-training objective, pre-trained on dileptonic ttbar events. Key findings include a 10-20% improvement in ttbar mass reconstruction, AUROC 0.877 for toponium discrimination, and AUROC 0.625 for initial-state classification, demonstrating strong new-effect discovery potential. The approach supports broad applicability to diverse collider processes and future physics discoveries.

Abstract

Bumblebee is a foundation model for particle physics discovery, inspired by BERT. By removing positional encodings and embedding particle 4-vectors, Bumblebee captures both generator- and reconstruction-level information while ensuring sequence-order invariance. Pre-trained on a masked task, it improves dileptonic top quark reconstruction resolution by 10-20% and excels in downstream tasks, including toponium discrimination (AUROC 0.877) and initial state classification (AUROC 0.625). The flexibility of Bumblebee makes it suitable for a wide range of particle physics applications, especially the discovery of new particles.

Figures (3)

• Figure 1: The embedding procedure for Bumblebee for a $\mathrm{t\bar{t}}$ dileptonic decay. The final embedded input given to Bumblebee is the unweighted sum of the particle vector embedding, PDG ID embedding, level type embedding, and mask status embedding.
• Figure 2: A) The receiver operating characteristic (ROC) curve for Bumblebee fine-tuned on discriminating toponium against $\mathrm{t\bar{t}}$. Two supervised models, DNN and Transformer, are shown for comparison. B) The ROC curve for Bumblebee fine-tuned on discriminating the initial state of $\mathrm{t\bar{t}}$. The positive class is the gluon-gluon initial state. Two supervised models, DNN and Transformer, are shown for comparison. C) The resolution of $m(\mathrm{t\bar{t}})$ given as the difference between the 16$^{th}$ and 84$^{th}$ percentiles of the $m(\mathrm{t\bar{t}})$ residuals ($P_{84} - P_{16}$) as a function of the true $m(\mathrm{t\bar{t}})$.
• Figure 3: The comparison of the best validation loss obtained on the pre-training objective when removing individual embeddings present in the input representation. The embeddings removed are labeled on the y-axis.