A Language Model for Particle Tracking

TL;DR

This work addresses the limited generalization of task-specific DL models for LHC particle tracking by introducing a tokenized detector representation and Training a BERT-based TrackingBERT on self-supervised tasks to learn latent detector embeddings. TrackingBERT achieves near-perfect masked-hit prediction (99.8% exact, 100% within 20 mm) and demonstrates robustness to track length, suggesting a path toward a foundational, transferable detector model. The approach provides a unified framework to apply language-model concepts to detector understanding and multi-task particle reconstruction, with potential for broader applicability in high-energy physics analysis.

Abstract

Particle tracking is crucial for almost all physics analysis programs at the Large Hadron Collider. Deep learning models are pervasively used in particle tracking related tasks. However, the current practice is to design and train one deep learning model for one task with supervised learning techniques. The trained models work well for tasks they are trained on but show no or little generalization capabilities. We propose to unify these models with a language model. In this paper, we present a tokenized detector representation that allows us to train a BERT model for particle tracking. The trained BERT model, namely TrackingBERT, offers latent detector module embedding that can be used for other tasks. This work represents the first step towards developing a foundational model for particle detector understanding.

Figures (3)

• Figure 1: (a) Layout of the TrackML detector. (b) Number of hits in one track, namely the sequence length in the figure.
• Figure 2: Distances between model predicted module location and the true module location.
• Figure 3: Model performance in predicting masked modules for different track length.