Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models

Chenfeng Xu, Shijia Yang, Tomer Galanti, Bichen Wu, Xiangyu Yue, Bohan Zhai, Wei Zhan, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka

TL;DR

The paper investigates transferring 2D image pretrained models to 3D point-cloud understanding by inflating 2D networks into 3D sparse conv nets and fine-tuning only input/output and normalization layers (FIP-IO/BN) or the entire network (FIP-ALL). It demonstrates that image priors significantly boost point-cloud classification and segmentation across ModelNet, S3DIS, and SemanticKITTI, achieving competitive results with limited finetuning and substantial data-efficiency gains (up to 10.0 top-1 in few-shot) and training speedups (up to 11.1x). The work also provides empirical evidence and a theoretical framing based on neural collapse to explain why cross-modal transfer is effective, highlighting the role of learned class-cluster structure and an adaptable adaptor. Overall, this approach shows that leveraging large-scale image pretraining can mitigate the bottleneck of point-cloud supervision and offers a practical pathway for rapid, data-efficient 3D understanding. The codebase is released to facilitate adoption and further exploration in cross-modal 3D perception.

Abstract

3D point-clouds and 2D images are different visual representations of the physical world. While human vision can understand both representations, computer vision models designed for 2D image and 3D point-cloud understanding are quite different. Our paper explores the potential of transferring 2D model architectures and weights to understand 3D point-clouds, by empirically investigating the feasibility of the transfer, the benefits of the transfer, and shedding light on why the transfer works. We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds. Specifically, we transfer the image-pretrained model to a point-cloud model by copying or inflating the weights. We find that finetuning the transformed image-pretrained models (FIP) with minimal efforts -- only on input, output, and normalization layers -- can achieve competitive performance on 3D point-cloud classification, beating a wide range of point-cloud models that adopt task-specific architectures and use a variety of tricks. When finetuning the whole model, the performance improves even further. Meanwhile, FIP improves data efficiency, reaching up to 10.0 top-1 accuracy percent on few-shot classification. It also speeds up the training of point-cloud models by up to 11.1x for a target accuracy (e.g., 90 % accuracy). Lastly, we provide an explanation of the image to point-cloud transfer from the aspect of neural collapse. The code is available at: \url{https://github.com/chenfengxu714/image2point}.

Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models

TL;DR

The paper investigates transferring 2D image pretrained models to 3D point-cloud understanding by inflating 2D networks into 3D sparse conv nets and fine-tuning only input/output and normalization layers (FIP-IO/BN) or the entire network (FIP-ALL). It demonstrates that image priors significantly boost point-cloud classification and segmentation across ModelNet, S3DIS, and SemanticKITTI, achieving competitive results with limited finetuning and substantial data-efficiency gains (up to 10.0 top-1 in few-shot) and training speedups (up to 11.1x). The work also provides empirical evidence and a theoretical framing based on neural collapse to explain why cross-modal transfer is effective, highlighting the role of learned class-cluster structure and an adaptable adaptor. Overall, this approach shows that leveraging large-scale image pretraining can mitigate the bottleneck of point-cloud supervision and offers a practical pathway for rapid, data-efficient 3D understanding. The codebase is released to facilitate adoption and further exploration in cross-modal 3D perception.

Abstract

3D point-clouds and 2D images are different visual representations of the physical world. While human vision can understand both representations, computer vision models designed for 2D image and 3D point-cloud understanding are quite different. Our paper explores the potential of transferring 2D model architectures and weights to understand 3D point-clouds, by empirically investigating the feasibility of the transfer, the benefits of the transfer, and shedding light on why the transfer works. We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds. Specifically, we transfer the image-pretrained model to a point-cloud model by copying or inflating the weights. We find that finetuning the transformed image-pretrained models (FIP) with minimal efforts -- only on input, output, and normalization layers -- can achieve competitive performance on 3D point-cloud classification, beating a wide range of point-cloud models that adopt task-specific architectures and use a variety of tricks. When finetuning the whole model, the performance improves even further. Meanwhile, FIP improves data efficiency, reaching up to 10.0 top-1 accuracy percent on few-shot classification. It also speeds up the training of point-cloud models by up to 11.1x for a target accuracy (e.g., 90 % accuracy). Lastly, we provide an explanation of the image to point-cloud transfer from the aspect of neural collapse. The code is available at: \url{https://github.com/chenfengxu714/image2point}.

Paper Structure

This paper contains 43 sections, 2 theorems, 26 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Point-Cloud Processing Models
  4. Pretraining in 2D and 3D Computer Vision
  5. Cross-Modal Transfer Learning
  6. Neural Collapse
  7. Converting a 2D Image Model to a 3D Point-Cloud Model
  8. Inflating a 2D ConvNet into a 3D sparse ConvNet.
  9. A note on image-to-video transfer.
  10. Empirical Evaluation
  11. Datasets.
  12. Can we transfer pretrained-image models to recognize point-clouds?
  13. Can image-pretraining benefit point-cloud recognition?
  14. Can image-pretrained models improve the data efficiency on point-cloud recognition?
  15. Can image-pretrained models accelerate point-cloud training?
  16. ...and 28 more sections

Key Result

Theorem 1

In the setting above. For any tuple $(f,\tilde{g}) \in \mathcal{F} \times \mathcal{G}$, we have: where $s(f,P_1,P_2)=p_2$ if $f \circ P_1$ and $f \circ P_2$ are spherically symmetric and $s(f,P_1,P_2)=1$ otherwise.

Figures (5)

  • Figure 1: We investigate the feasibility of converting pretrained 2D image models to 3D point-cloud models. For example, with filter inflation and finetuning the input, output (classifier for classification task and decoder for semantic segmentation task), and normalization layers, the transformed 2D ConvNets are able to deal with point-cloud classification, indoor, and driving scene segmentation.
  • Figure 2: a) the left figure shows the trainable parameters ratio w.r.t top-1 accuracy on ModelNet 3D Warehouse dataset. b) the right figure shows the performance of FIP-IO and FIP-IO+BN on top of ResNet50 pretrained on different datasets.
  • Figure 3: Comparing the validation accuracy of training-from-scratch and FIP-ALL on ModelNet 3D. We report the validation accuracy during training. We compare the results between training-from-scratch and fine-tuning the whole network (FIP-ALL) with pretraining ResNet18, ResNet50, and ResNet152, on the ImageNet1K dataset.
  • Figure 4: tSNE visualization and class-distance normalized variance on ModelNet 3D Wharehouse dataset. FIP-IO+BN on ImageNet1K/21K are the same models in Fig. \ref{['fig:exp_pfip']}. CDNV is computed for the fine-tuned model on the train and test data of the point-cloud domain.
  • Figure 5: Visualization of filter inflation along different axis.

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2
  • proof