Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics

TL;DR

The paper tackles real-time processing of massive scientific point clouds by introducing HEPT, a locality-sensitive hashing-based Efficient Point Transformer that embeds explicit local inductive bias in the attention kernel. It provides a rigorous error-computation analysis showing that Random Fourier Features under subquadratic budgets are outperformed by LSH, and that combining OR & AND LSH yields exponential error decay at near-linear computational cost. HEPT consequently achieves state-of-the-art accuracy and massive speedups (up to 203x) on two high-energy physics tasks (tracking and pileup mitigation) without requiring graph construction, highlighting its practical value for large-scale geometric deep learning in physics. The work also demonstrates solid scalability and robustness to hyperparameter choices, and offers a publicly available implementation for broader adoption in scientific domains.

Abstract

This study introduces a novel transformer model optimized for large-scale point cloud processing in scientific domains such as high-energy physics (HEP) and astrophysics. Addressing the limitations of graph neural networks and standard transformers, our model integrates local inductive bias and achieves near-linear complexity with hardware-friendly regular operations. One contribution of this work is the quantitative analysis of the error-complexity tradeoff of various sparsification techniques for building efficient transformers. Our findings highlight the superiority of using locality-sensitive hashing (LSH), especially OR & AND-construction LSH, in kernel approximation for large-scale point cloud data with local inductive bias. Based on this finding, we propose LSH-based Efficient Point Transformer (HEPT), which combines E$^2$LSH with OR & AND constructions and is built upon regular computations. HEPT demonstrates remarkable performance on two critical yet time-consuming HEP tasks, significantly outperforming existing GNNs and transformers in accuracy and computational speed, marking a significant advancement in geometric deep learning and large-scale scientific data processing. Our code is available at https://github.com/Graph-COM/HEPT.

Figures (8)

• Figure 1: Pipeline of HEPT. Elements that share the same color represent points from the same local neighborhood. HEPT employs OR & AND LSH to minimize noise caused by individual hash functions. HEPT also integrates point coordinates as extra AND LSH codes for query-key alignment, maintaining computational regularity without compromising accuracy.
• Figure 2: The error-computation tradeoff from numerical experiments. OR & AND LSH decreases the error exponentially with near-linear complexity, validating our analysis.
• Figure 3: The above shows how to obtain AND hash code $T^{(i)}$ with $m_2 = 3, B_{ij} = 2$. Points are assumed to be pre-sorted based on their raw hash values with $\min(L^{(i, 1)})=0$.
• Figure 4: Illustrations of the two HEP tasks.
• Figure 5: Inference and training costs per point cloud.

Theorems & Definitions (15)

• Definition 3.0: Bounded-Support Kernels
• Theorem 3.1: $\epsilon-F$ Tradeoff of RFF
• Theorem 3.2: $\epsilon-F$ Tradeoff of OR-only E$^2$LSH
• Theorem 3.3: $\epsilon-F$ Tradeoff of OR & AND E$^2$LSH
• Definition 5.0: Bounded-Support Kernels
• Theorem 5.1: $\epsilon-F$ Tradeoff of RFF
• proof
• Lemma 5.1
• proof
• Lemma 5.2