Adaptive Point Transformer

TL;DR

The paper tackles the scalability challenge of point cloud transformers by introducing AdaPT, which uses learnable drop predictors to adaptively downsample tokens during inference. A budget mechanism enables flexible control over computational cost without retraining, supported by a differentiable Gumbel-Softmax sampling strategy and a regularization term to encourage token elimination. Empirical results on ModelNet40 show AdaPT achieving competitive accuracy across varying budgets while reducing FLOPs, underscoring its practical utility for resource-constrained scenarios. The work also demonstrates that learnable token selection can preserve performance while enabling real-world deployment of large-scale point cloud models.

Abstract

The recent surge in 3D data acquisition has spurred the development of geometric deep learning models for point cloud processing, boosted by the remarkable success of transformers in natural language processing. While point cloud transformers (PTs) have achieved impressive results recently, their quadratic scaling with respect to the point cloud size poses a significant scalability challenge for real-world applications. To address this issue, we propose the Adaptive Point Cloud Transformer (AdaPT), a standard PT model augmented by an adaptive token selection mechanism. AdaPT dynamically reduces the number of tokens during inference, enabling efficient processing of large point clouds. Furthermore, we introduce a budget mechanism to flexibly adjust the computational cost of the model at inference time without the need for retraining or fine-tuning separate models. Our extensive experimental evaluation on point cloud classification tasks demonstrates that AdaPT significantly reduces computational complexity while maintaining competitive accuracy compared to standard PTs. The code for AdaPT is made publicly available.

Figures (8)

• Figure 1: Architecture of the AdaPT model. The embedding module consists of an Absolute-Relative positional embedding Yang2019modeling. The transformer blocks are paired with the proposed drop predictors. Finally, the representation is processed by a classifier head which outputs a prediction.
• Figure 2: Scheme of the drop predictor module architecture.
• Figure 3: Drop predictor modules usage in the PCT.
• Figure 4: Visualization of the kept tokens along the model's layers for a few representative examples.
• Figure 5: The budget parameter selects the set of drop predictors to be used in the classification of a point cloud. This parameter also selects a specific regularization term that is used to train the corresponding drop predictor set.